OVERVIEW: cvae module - convolutional variational auto encoder

TODO:clean up this preamble section preamble: This is a project initiated while an Insight Data Science fellow. It grew out of my interest in making data driven tools in the fashion/retail space I had most recently been working. The original over-scoped idea was to make a shoe desighn tool which could quickly develop some initial sneakers based on choosing some examples, and some text descriptors. Designs are constrained by the "latent space" defined (discovered?) by a database of shoe images. However, given the 3 week sprint allowed for development, I pared the tool down to a simple "aesthetic" recommender for sneakers, using the same idea of utilizing an embedding space defined by the database fo shoe images.

Most of the code is derived from the work of @EtieeneT. (e.g.: TabularData https://github.com/EtienneT/TabularVAE and later https://github.com/EtienneT/vae )

sneaker image encoding via various flavors of auto-encoder.

Auto Encoder: AE
- ~~linear encoder / decoder~~
- convolutional encoder / decoder
  - encoders
    - pretrained (e.g. resnet, mobilenet_v2)
    - 'vanilla' convolutional
    - 'vanilla' ResBlock
  - decoders
    - 'vanilla' convolutional
    - 'vanilla' ResBlock
- Linear Latent representation (LatentLayer)
Variational- Auto Encoder: VAE or 𝜷-VAE
- Variational "bottleneck": "reparameterazation trick" for two variable linear latents
  - 𝜷-VAE: by increasing the relative strenght of error signal of the KLD through a beta variational parameter we can get better formed latent spaces which are somewhat "disentangled". That is have some semantic or meaningful structure in the representation.
- MMD- variational autoencoder uses a Linear Latent and MMD as the retularizer
  - The KLD has some problems and "over-regularizes" or simply isn't strong enough compared to the Maximum Mean Discrepancy (MMD) loss and tends to converge to a somewhat degenerate solution
  - MD regularization of the latent space has great advantage but at a computational expense

FUTURE EXTENSIONS:

-  model based data cleaning (widget module?)
- GAN finetuning?
- crappify general pattern
- throw out based on inspection of high loss
- try mixed labels?  things that are >50% sneakers included???

Using fastai V2: data pipelining

Datablock API

Load the saved merged database, and set the seeds. And doublecheck our data is where we expect.

df = prep_df_for_datablocks(df)

FastAI "block" for autoencoder

Next we need our own version of ReadTabBatch that will return our inputs

Don't forget to set n_inp=1. Otherwise the default to make the input to 1-len(blocks). Also note that the FeatsResize is used to avoid the random jittering from resize during training. Only the very narrow batch augmentations will be used.

creating the VAE

Variational Auto-Encoder for fastai

I'm going to use a generic convolutional net as the basis of the encoder, and its reverse as the decoder. This is a proof of concept for using the fastai framework, and will experiment with pre-trained resnet and MobileNet_v2 later. I'd like to use the MobileNet_v2 as a direct link ot the "SneakerFinder" tool which motivated this experiment. [see SneakerFinder]

A variational "module" will sit between the encoder and decoder as the "Bottleneck". The Bottleneck will map the resnet features into a latent space (e.g. ~100 dimensions) represented of standard normal variables. The "reparameterization trick" will sample from this space and the "decoder" will generate images.

Finally a simple "decoder" will sample from the variational latents space and be trained to reconstruct the images.

The intention is the latent space can be used to generate novel sneaker images.

AE: deterministic AutoEncoder ( non- variational)

Although we give up the original utility we are going for -- creating new sneakers via the latent space -- having otherwise equivalent non-variational autoencoders for reference will be great. Furthermore, this latent space representation will be amenable to a MMD regularization later on. This will be useful to avoid some of the limitiations of the KLD as a regularizer (overestimation of variance, and some degenerate convergences). Its sort of hack-ey but keeping the tooling equivalent to the betaVAE will ultimatel give some advantages.

It is convenient to avoid the class wrappers to simplify the param splitting for freezing the pre-trained arcitecture.
We could enumerate the class layers and return sequential, but simply making some functions to put the layers togeter is better.

Loss

These inherit from Module. (fastai's wrapper on nn.Module). The base class AELoss initializes wiht a batchmean, alpha, and useL1 parameters to set how the loss will be aggregated and regularized. For the basice AutoEncoder we'll regularize the latent with a L1 to keep the magnitudes from exploding.
{% include note.html content='Here batchmean means we will divide the loss (either L1 or L2 depending on useL1 flag) by which uses reduction=’sum’ by the batch size. This technically makes it a cost computed for each batch. This same convention will be used later for KL-Divergence and MMD latent regularizers.' %}

Metrics

These are called by the recorder callback and collect quantities to track training. I made a simple meta class so I don't have to repreat the reset and mean value property for all the metrics.

These are some helpers for computing the KL Divergence as a function and a Module

metrics helpers

Callback - AnnealedLoss

The AnnealedLoss Callback basically injects a kl_weight parameter into the loss so we can start training without the full KLD regularization for the beta-VAE version.

The fastai Learner class does the training loop. It took me a little digging into the code to figure out how Metrics are called since its not really stated anywhere in the documentation (Note: create PR for fastai for extra documentation on Metrics logic). By default one of the key Callbacks is the Recorder. It prints out the training summary at each epoch (via ProgressCallBack) and collects all the Metrics. Which by default only loss is a train_met and others are valid_met. The Recorder resets (maps reset() to all mets) the metrics before_train and before_valid. The Recorder maps accumulate() to the metrics on after_batch. Finally

AnnealedLossCallback will inject the latent mu and logvar and a kl_weight variable into our loss. The mu and logvar will be used to compute the KLD. The kl_weight is a scheduled weighting for the KLD. You can see the schedule graph of the parameter. At the beginning it will be 0, thus the KLD part of the loss will get ignored. So during 10% of training, we will fit a normal auto-encoder. Then gradually for 30% of trainning, increase kl_weight to 1 and then remain there for the remaining training time so that the auto encoder now becomes full variational. The way this callback is done, the loss will receive this parameter, but not the model.

n_epochs = 10
f_init = combine_scheds([.1, .7, .2], [SchedNo(0,0),SchedCos(0,1), SchedNo(1,1)])
# f = combine_scheds([.8, .2], [SchedCos(0,0), SchedCos(0,.5)])
p = torch.linspace(0.,1.,100)
pp = torch.linspace(0.,1.*n_epochs,100)

plt.plot(pp,[f_init(o) for o in p])

[<matplotlib.lines.Line2D at 0x7f4e01251a00>]

{% include warning.html content='Avoid using early stopping because the AnnealedLossCallback will make the loss grow once the KL divergence weight kicks in. ' %}

I want to note something here that was a little confusing to me: params(model) is a builtin fastai PyTorch.core function which returns all of the parameters of the modules. i.e.

def params(m):
    "Return all parameters of `m`"
    return [p for p in m.parameters()]

The toplevel fastai core functions with simple names that almost match class attributes was one of my biggest stumbling blocks in getting acquainted with the fastai v2 API. (The other is the documentation which is autogenerated by the fastdev frameworks from their development noteboooks. More on that struggle and my tips if that is troblesome for you later (here). {% include note.html content='that it is crucial that you don’t freeze the batch norm layers. The bn_splitter collects out all the batchnorm layers. The simple splitting we do only freezes the encoder and leaves the latent layers (i.e. VAE or linear encoding bottlenedck) and the decoder in a parameter group with the batchnorm layers.' %}

parameter `Splitters`

{% include warning.html content='there are two completely different splitters in the FastAI API. This splitter groups the model parameters into groups for freezing and for progressive learning rates. (The other one is splits data into train and validate. I got imminiently confused when I first started with the API by this.' %}

TODO:more sophisticated parameter splitting to enable progressive learning rates

other encoder and decoder types

MobileNet_v2 as the encoder, as a continuation of the original Sneaker Finder

simple bowtie convolutional encoder / decoder (Mimics the GOAT medium blog)

Architecture Hyperparameters:- Latent Size (research default 256, production default 32) - Filter Factor Size (research default 16, production default 32)

Latent Linear Hidden Layer Size (research default 2048, production default 1024)

The encoder architecture is as follows with research defaults from above:

Input 3x128x128 (conv2d block [conv2d, batchnorm2d, relu])

16x64x64 (conv2d block [conv2d, batchnorm2d, relu])

32x32x32 (conv2d block [conv2d, batchnorm2d, relu])

64x16x16 (conv2d block [conv2d, batchnorm2d, relu])

128x8x8 (conv2d block [conv2d, batchnorm2d, relu])

Flatten to 8192

2048 (linear block [linear, batchnorm1d, relu])

Split the 2048 dimension into mu and log variance for the parameters of the latent distribution

Latent mu size 256 (linear layer only with bias)

Latent logvar size 256 (linear layer only with bias)

In the middle here you can break out the BCE and KLD loss for the final loss term and use the standard reparam trick to sample from the latent distribution.

Decoder architecture an exact mirror

Input 256

2048 (linear block [linear, relu])

8192 (linear block [linear, batchnorm1d, relu])

reshape (128x8x8)

64x16x16 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])

32x32x32 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])

16x64x64 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])

3x128x128 (conv2d transpose [convtranspose2d, sigmoid]

For weight initialization I used a normal distribution centered at zero with 0.02 set for the stddev. Optimizer: Adam with default parameters, if I were to do it over again I'd spend more time here understanding the learning dynamics. The dataset was about ~10,000 with a 70/20/10 split, batch size 64, over 120 epochs, with a learning schedule to reduce when the loss started to plateau. No crazy image augmentation just resizing and standards flips. I used the ANN package Annoy to do the NN search for prod, normalizing the embeddings and using the cosine similarity, ANN factor was 128 for num_trees.

MMD regularized VAE where the latents are drawn from a

TODO:Ranger optimizer might really help .. test

We can also use the transfer learning VAE tooling we previously built. We just need to create the convolutional encoder and pass it in... Note that we don't have a pre-trained option, so DON'T FREEZE!

Now just wrap that simple conv block architecture into a builder. And a meta-wrapper to let us call the conv_encoder and pre-trained options with the same function. (I'll also put the get_pretrained_parts function here now even though we won't use it till the next section, so that we can make the get_encoder_parts generic wrapper handle both properly.)

##   


latent_dim = 128

# equalize KLDiv wrt errors per pixel
alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha /= 20  # 5% retularizer

batchmean = True 
useL1 = False
hidden_dim = None
# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),  ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

metrics = default_AE_metrics(alpha,batchmean,useL1)

block = get_ae_DataBlock(aug=True)
batch_size = 64
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'

vae = AE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = AELoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,

{% include note.html content='The to_fp16() callbacks work but increasing the batch size doesn’t really speed things up.' %}

latent_dim = 128

# equalize KLDiv wrt errors per pixel
alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha /= 20  # 5% retularizer

batchmean = True 
useL1 = False
hidden_dim = None
# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),  ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

metrics = default_AE_metrics(alpha,batchmean,useL1)

block = get_ae_DataBlock(aug=True)
batch_size = 64
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'

vae = AE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = AELoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
learn = learn.to_fp16()

constructing VAE with Module Class Layers

For several of the decoder and "sampler" layers I might want to turn off the nonlinearity to give us more reasonable "gaussian" outputs to the Variational layer and the generated image which will is compared with the ImageNetStats batch-normalized image.

IMPORTANT VAE TIP!!! Make sure NOT to use batch normalization and non-linearity in the linear layers of the VAE. The normalization will affect the representation and the KLD constraints.

simple BVAE class from VAE layer class components

This creates a pair of latents from which we can perform the "resample" trick. {% include note.html content='this is a $\beta$-VAE (hence BVAE because we have a weighting factor for the KL Divergence regularazation factor which acts as a Legrangian).' %}

Putting it all together gives us our VAE! Note that we'll pass in the "parts" of the encoder for ease of using pretrained (or not) architectures. The model name will correspond to the architecture of the encoder via name.

Note that the BVAE can simply inherit from the AE class we defined above. Really the only difference in the __init__ function is that a VAELayer which performs the reparameterization trick replaces the AElayer as self.bn

A nice wrapper for building the encoder parts will be handy.

Sweet, we've verified the arcitecture works, but we need to train it with a loss that constrains the variational layers with the KL Divergence. Otherwise the simple MSE will diverge.

VAE Loss functions classes

This simply adds a KLD regularizer to the latent space defined by two rank-1 tensors defining gaussian-prior latents. i.e. a mean ($\mu$) and standard deviation ($\sigma$).
{% include note.html content='for convenience and numeric stability the $\sigma$ is representated as a $\log(\sigma^{s})$ so the tensors are called mu and logvar' %}

Here's how we put everything together.

latent_dim = 128

alpha = 5
batchmean = True
useL1 = False
hidden_dim = None


cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]

metrics = default_VAE_metrics(alpha,batchmean,useL1)

We can use this callback if we want to save the model at every epoch. Which could be super useful if we were able to actually overfit our model.

'SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True)'

# NOTE: lf_finder does NOT work correctly with our annealed kl_weight... 

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.0002511886414140463,
 0.0003311311302240938,
 0.00029115988581907004,
 0.00028840304003097117)

{% include note.html content='The CallBacks need to be updated for the KL loss annealing schedule as we tune. We want to turn the kl_weight to 1.0 for instance when using the learning rate finder learn.lr_find(), and after an initial burn in where the KL_loss term gradually ramps in, setting the kl_weight to stay at unity will be useful as we separately turn the learning rate (e.g. fit_one_cycle or fit_flat_cosine.' %}

#put in the annealied KL_weight...
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler

learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f4e019d9e20>

n_epochs = 5

#learn.fit_one_cycle(n_epochs,lr_max= lr1)
learn.fit_one_cycle(n_epochs)
learn.show_results()

The vae with pretrained resnet encoder seems to train to a much better end-point if we keep the resnet frozen. Hence the commented out learn.unfreeze() below.

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f4e019d9e20>

epochs = 20

#learn.fit_one_cycle(epochs, lr_max= 1e-3)
learn.fit_flat_cos(epochs,pct_start=.05)

learn.show_results()

"Vanilla" convolutional beta-VAE

latent_dim = 128

# equalize KLDiv wrt errors per pixel
#  alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha = 5
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight': default_KL_anneal_in() })]

metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'
vae = BVAE(get_encoder_parts(arch), hidden_dim=None,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

MMD-VAE

The MMD replace latent regularization term in loss_fn (KLD) with Maximal Meanm Discrepancy.

We'll make an MMDVAE class to keep things declarative, but its really just an AE. i.e. a linear latent layer.

Additional background on MME from [https://github.com/Saswatm123/MMD-VAE]:

Maximum Mean Discrepancy Variational Autoencoder, a member of the InfoVAE family that maximizes Mutual Information between the Isotropic Gaussian Prior (as the latent space) and the Data Distribution.\ Short explanation:The traditional VAE is known as the ELBO-VAE, named after the Evidence Lower Bound used in its objective. The ELBO suffers from two problems: overestimation of latent variance, and uninformative latent information.\The latter is because one of the objective's terms is the KL-Divergence between the Gaussian parameterized by the encoder and the Standard Isotropic Gaussian. This dissuades usage of the latent code, so that the KL-Divergence term is allowed to fall even further. It is important to note that the KL-Divergence should never truly reach zero, as that means the encoder is not learning useful features and cannot find feature locality, and the decoder is just randomly sampling from Standard Gaussian noise.\ The overestimation of variance results from the KL-Divergence term not being strong enough to balance against the Reconstruction Error, and thus the Encoder prefers to learn a multimodal latent distribution with spread apart means, leading to low training error as it overfits, but low quality samples as well, as the sampling distribution is assumed to be a Standard Isotropic Gaussian. One effort to mitigate this effect is the Disentangled Variational Autoencoder, which simply raises the weight on the KL-Divergence term. However, this increases the problem stated in the paragraph above since it further penalizes using the latent code.\ For more detailed explanations, I used these resources to learn, in order of usefulness to me:\
- https://arxiv.org/pdf/1706.02262.pdf \
- http://ruishu.io/2018/03/14/vae/ \
- http://approximateinference.org/accepted/HoffmanJohnson2016.pdf \
- https://ermongroup.github.io/blog/a-tutorial-on-mmd-variational-autoencoders/ \
- http://bjlkeng.github.io/posts/variational-bayes-and-the-mean-field-approximation/ \
- https://ermongroup.github.io/cs228-notes/inference/variational/ \

Vanilla MMD VAE

Simply call our get_encoder_parts with arch='vanilla' in the MMDVAE builder.

latent_dim = 128

cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
alpha = 20

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,

ResNet Encoder MMDVAE

We build these by passing resnet18 (not 'resnet18') to get_encoder_parts helper for the parts to init the MMDVAE.

First a traditional "fine_tune" type of training:

latent_dim = 128

cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
learn.freeze()

learn.freeze()
n_epoch = 20
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(n_epoch)#,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

n_epoch = 20
learn.unfreeze()
#learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.7)
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

vanilla-resnet MMD VAE

no freeze

We build these by passing resnet18 (not 'resnet18') to get_encoder_parts helper for the parts to init the MMDVAE.

{% include note.html content='our architecture trains best when simply starting with the pretrained weights. Trying to "fine_tune" by training the backend on a frozen resnet and then unfreezing doesn’t work with the parameter groupings from the AE_split. The learn.unfreeze() doesn’t actually do anythign (unfrozen is default) but makes is clear we are un -frozen' %}

latent_dim = 128

cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]


alpha = 10

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
    
    
learn.unfreeze()

vanilla-resnet MMD-VAE

ResBlocks replacing Conv2d in encoder AND decoder.

'Vanilla' encoder architecture made with ResBlock layers instead of ConvLayer layers and Mish activation.

Finally we can use the ResBlockBVAE which instantiates a ResBlock decoder to optimize the architecture. This is following the fastAI API lessong from the "bag of tricks" ResNet paper (CITATION), which is a general true-ism which could be glibly states as: "replacing a Conv with a ResBlock always gets you better results". The Class constructor ResBlockAE takes isVAE to switch between AELayer and LatentLayer latents.

TODO:update AE class to make a VAE or AE based on the isVAE switch

latent_dim = 128
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE,isVAE= True)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

next steps.

visualize activations in encoder decoder
improve training with progressive learning rates
G

export / provenance

128 latents, alpha=10

latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.03981071710586548,
 0.18547092080116273,
 0.11481534689664841)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent128-resblock-alpha10_2021-03-24_10.15.31.pth')

128 latents, alpha=20

latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 20
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.04786301031708717,
 0.18949706740677358,
 0.1258925348520279)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent128-resblock-alpha20_2021-03-24_11.36.20.pth')

64 latents, alpha=10

latent_dim = 64

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.05754399299621582,
 0.1943375587463379,
 0.13803842663764954)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent64-resblock-alpha10_2021-03-24_12.57.06.pth')

64 latents, alpha = 20

latent_dim = 64

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.03981071710586548,
 0.18547092080116273,
 0.11481534689664841)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent64-resblock-alpha10_2021-03-24_14.17.51.pth')

vanilla-resnet beta-VAE

class ResBlockBVAE(BVAE):
    """
    simple VAE with a _probably_ pretrained encoder 
    """

    def __init__(self,enc_parts,hidden_dim=None, latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE):
       
        """
        inputs:  
            enc_arch (pre-cut / pretrained)
            enc_dim
            latent_dim
            hidden_dim
            im_size,out_range
        """
        enc_arch,enc_feats,name = enc_parts

        # encoder
        #  arch,cut = xresnet18(pretrained=True),-4
        #  enc_arch = list(arch.children())[:cut]
        
        BASE = im_size//2**5
        enc_dim = enc_feats * BASE**2  # 2**(3*3) * (im_size//32)**2 #(output of resneet) #12800

        self.encoder = build_AE_encoder(enc_arch,enc_dim=enc_dim, hidden_dim=hidden_dim, im_size=im_size)

        in_dim = enc_dim if hidden_dim is None else hidden_dim

        # VAE Bottleneck
        self.bn = VAELayer(in_dim,latent_dim)     

        #decoder
        self.decoder = build_ResBlockAE_decoder(hidden_dim=hidden_dim, latent_dim=latent_dim, im_size=im_size,out_range=out_range)

        store_attr('name,enc_dim, in_dim,hidden_dim,latent_dim,im_size,out_range') # do i need all these?




#  THESE ARE INHERITED..
#     def decode(self, z):    
#         z = self.decoder(z)
#         return z
    
#     def reparam(self, h):
#         return self.bn(h)

#     def encode(self, x):
#         h = self.encoder(x)
#         z, mu, logvar = self.reparam(h)
#         return z, mu, logvar

#     def forward(self, x):
#         z, mu, logvar = self.encode(x)
#         x_hat = self.decode(z)
#         latents = torch.stack([mu,logvar],dim=-1)
#         return x_hat, latents # assume dims are [batch,latent_dim,concat_dim]

128 latents, alpha=5

latent_dim = 128
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.0001737800776027143,
 0.0012022644514217973,
 0.0006880222645122558,
 0.0004570883756969124)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8e9203d0>

# the defaults are pretty good for now
n_epochs = 10

#learn.fit_one_cycle(freeze_epochs1,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs) #, lr=1e-4,pct_start=0.5)
learn.fit_one_cycle(n_epochs)#, lr_max= base_lr)

learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8e9203d0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(6.309573450380412e-08,
 1.5848931980144698e-06,
 8.239944662591369e-07,
 3.162276129842212e-07)

base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final=1000)#,lr=1e-4)
learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

x = 1

128 latents, alpha=10

latent_dim = 128
alpha = 10 # doubled because latent is half?
batchmean = True
useL1 = False



# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.00043651582673192023,
 0.002511886414140463,
 0.0014742011204361915,
 0.001047128695063293)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8e940af0>

# # the defaults are pretty good for now
n_epochs = 10

learn.fit_one_cycle(n_epochs)#, lr_max= base_lr)
# #learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
# learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

# learn.show_results()

# # the defaults are pretty good for now
# n_epochs = 10

# learn.fit_one_cycle(10)#, lr_max= base_lr)
# #learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
# #learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

# learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8e940af0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(6.918309954926372e-05,
 1.4454397387453355e-05,
 4.181874846835854e-05,
 3.16227633447852e-05)

epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final = 1000)#,lr=1e-4)
learn.show_results()

learn.lr

0.001

# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

# learn.save(filename)
# learn.export(f'{filename}.pkl')

# base_lr = 1e-5# gmlr #/= 2
# epochs = 50

# #learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
# #learn.fit_one_cycle(epochs, lr_max= 1e-3)
# #learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
# learn.fit_flat_cos(epochs)
# learn.show_results()

# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

# learn.save(filename)
# learn.export(f'{filename}.pkl')

# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"
# filename = "BVae-POST-1CYCLE10-latent128-resblock-alpha10_2021-03-24_21.33.02"
# learn.load(filename)
# #epochs = 5
# epochs = 10
# learn.fit_one_cycle(epochs, lr_max=.001)
# #learn.fit_flat_cos(epochs,lr=.0015,pct_start=.5,div_final=1000.0)
# #learn.fit_one_cycle(epochs,lr_max=5e-3,pct_start=0.5,div_final=100000) # gets down to ~4500 loss in 10
# learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

64 latents, alpha=5

latent_dim = 64
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False


# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.0002511886414140463,
 0.0030199517495930195,
 0.0016355701955035329,
 0.0008709632093086839)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8f2804f0>

n_epochs = 10

learn.fit_one_cycle(n_epochs)#,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8f2804f0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(5.754399353463669e-07,
 2.2908675418875646e-06,
 1.4331537386169657e-06,
 1.1481544106572983e-06)

base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
#learn.fit_flat_cos(epochs,lr=1e-4)
learn.fit_flat_cos(epochs, div_final=1000.0)#,lr=1e-3)
learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

64 latents, alpha=10

latent_dim = 64
alpha = 10 # doubled because latent is half?
batchmean = True
useL1 = False


# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.00043651582673192023,
 0.00363078061491251,
 0.002033648220822215,
 0.0012589251855388284)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8f2327f0>

# the defaults are pretty good for now
n_epochs = 10

learn.fit_one_cycle(n_epochs)#,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8f2327f0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(6.309573450380412e-08,
 1.9054607491852948e-06,
 9.842782418445494e-07,
 3.467365559117752e-07)

base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final=1000.)
learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

CODE graveyard

class LatentTuple(fastuple):
    "Basic type for tuple of tensor (vectors)"
    _show_args = dict(s=10, marker='.', c='r')
    @classmethod
    def create(cls, ts): 
        if isinstance(ts,tuple):
            mu,logvar = ts
        elif ts is None:
            mu,logvar = None,None
        else:
            mu = None
            logvar = None
            
        if mu is None: mu = torch.empty(0)
        elif not isinstance(mu, Tensor): Tensor(mu) 
        
        if logvar is None: logvar = torch.empty(0)
        elif not isinstance(logvar,Tensor): Tensor(logvar)
            
        return cls( (mu,logvar) ) 
        
    def show(self, ctx=None, **kwargs): 
        mu,logvar = self
        if not isinstance(mu, Tensor) or not isinstance(logvar,Tensor): return ctx

        title_str = f"mu-> {mu.mean():e}, {mu.std():e}  logvar->{logvar.mean():e}, {logvar.std():e}"
    
        if 'figsize' in kwargs: del kwargs['figsize']
        if 'title' in kwargs: kwargs['title']=title_str
        if ctx is None:
            _,axs = plt.subplots(1,2, figsize=(12,6))
            x=torch.linspace(0,1,mu[0].shape[0])
            axs[0].scatter(x, mu[:], **{**self._show_args, **kwargs})
            axs[1].scatter(x, logvar[:], **{**self._show_args, **kwargs})
            ctx = axs[1]
        
        ctx.scatter(mu[:], logvar[:], **{**self._show_args, **kwargs})
        return ctx
    
# could we do a typedispatch to manage the transforms...?
def VAETargetTupleBlock(): 
    return TransformBlock(type_tfms=VAETargetTuple.create, batch_tfms=IntToFloatTensor)

def LatentTupleBlock(): 
    return TransformBlock(type_tfms=LatentTuple.create, batch_tfms=noop)
    

# class TensorPoint(TensorBase):
#     "Basic type for points in an image"
#     _show_args = dict(s=10, marker='.', c='r')

#     @classmethod
#     def create(cls, t, img_size=None)->None:
#         "Convert an array or a list of points `t` to a `Tensor`"
#         return cls(tensor(t).view(-1, 2).float(), img_size=img_size)

#     def show(self, ctx=None, **kwargs):
#         if 'figsize' in kwargs: del kwargs['figsize']
#         x = self.view(-1,2)
#         ctx.scatter(x[:, 0], x[:, 1], **{**self._show_args, **kwargs})
#         return ctx

latent_dim = 128
dropout = .2
im_size = IMG_SIZE
n_blocks = 5        
nfs = [3] + [2**i*n_blocks for i in range(n_blocks+1)] 
nfs.reverse()
# decoder = nn.Sequential(
#             nn.Linear(latent_size, 16),
#             UnFlatten(4),
#             ResBlock(1, 3, 4, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, 4, 8, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, 8, 16, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             #nn.AdaptiveAvgPool2d((3,im_size, im_size)) 
# )





n_blocks = 5
hidden_dim = 2048
out_range = [-1,1]
tst = nn.Sequential(
            nn.Linear(latent_dim,hidden_dim), #nn.Linear(latent_dim, 16)
            nn.Linear(hidden_dim,im_size*n_blocks*n_blocks), #nn.Linear(latent_dim, 16)
            ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[4], nfs[5], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[5], nfs[6], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            #nn.Dropout2d(dropout),
            #nn.Upsample(scale_factor=2),            #
            #nn.AdaptiveAvgPool2d((3,im_size, im_size)),
            SigmoidRange(*out_range), #nn.Sigmoid()
)
tst,nfs

inp=    torch.randn((32,latent_dim))
#last_size = model_sizes(tst ) #[-1][1]
#num_features_model(tst)
#last_size
#nfs
tst(inp).shape,nfs
#last_size

z_dim = 100
enc = nn.Sequential(
    ResBlock(1, 1, 16, act_cls=nn.ReLU, norm_type=None),
    nn.MaxPool2d(2, 2),
    ResBlock(1, 16, 4, act_cls=nn.ReLU, norm_type=None),
    nn.MaxPool2d(2, 2),
    Flatten()
)
# torch.Size([32, 1, 28, 28])
# torch.Size([32, 16, 28, 28])
# torch.Size([32, 16, 14, 14])
# torch.Size([32, 4, 14, 14])
# torch.Size([32, 4, 7, 7])
# torch.Size([32, 196])

latent_size = 100

enc = nn.Sequential(
            ResBlock(1, 3, 5, stride=2, act_cls=Mish),#  1->3
            ResBlock(1, 5, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 1, stride=2, act_cls=Mish),
            Flatten(),
            nn.Linear(400, latent_size) # 16->400
        )
#  torch.Size([32, 1, 28, 28])
# torch.Size([32, 5, 14, 14])
# torch.Size([32, 5, 7, 7])
# torch.Size([32, 1, 4, 4])
# torch.Size([32, 16])
# torch.Size([32, 4])       
inp=    torch.randn((32,3,160,160))
for ii in range(0,8):
    print(enc[:ii](inp).shape)

z = enc(inp)

dropout=0
dec = nn.Sequential(
            nn.Linear(latent_size, 16),
            UnFlatten(4),
            ResBlock(1, 1, 4, act_cls=Mish), #4->5
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 4, 8, act_cls=Mish),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 8, 16, act_cls=Mish),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(16, 3, 3), #1->3
            #nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((28, 28)),
            nn.Sigmoid()
        )

# torch.Size([32, 4])
# torch.Size([32, 16])
# torch.Size([32, 1, 4, 4])
# torch.Size([32, 4, 4, 4])
# torch.Size([32, 4, 8, 8])
# torch.Size([32, 8, 8, 8])
# torch.Size([32, 8, 16, 16])
# torch.Size([32, 16, 16, 16])
# torch.Size([32, 16, 32, 32])
# torch.Size([32, 1, 30, 30])
# torch.Size([32, 1, 28, 28])


for ii in range(0,12):
    print(dec[:ii](z).shape)

n_blocks = 5
BASE = im_size//2**5        
nfs = [3]+[(2**i)*BASE for i in range(n_blocks)] 
n = len(nfs)
hidden_dim = 2048

BASE = im_size//2**5
        # encoder
in_dim = nfs[-1] * BASE**2 
        
modules =  [ResBlock(1, nfs[i],nfs[i+1], 
                          stride=2, act_cls=Mish)  for i in range(n - 1)]    
# enc =  nn.Sequential(
#                 ConvLayer(nfs[0],nfs[1],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[1],nfs[2],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[2],nfs[3],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[3],nfs[4],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[4],nfs[5],ks=5,stride=2,padding=2),
#                 Flatten(),
#                 LinBnDrop(in_dim,hidden_dim,bn=True,p=0.0,act=nn.ReLU(),lin_first=True)
#             )

enc =  nn.Sequential(*modules,
                Flatten(),
                LinBnDrop(in_dim,hidden_dim,bn=True,p=0.0,act=nn.ReLU(),lin_first=True)
            )

nfs.reverse()

print(nfs)
#last_size = model_sizes(enc, size=(28,28))[-1][1]



    
encoder = nn.Sequential(enc, nn.Linear(hidden_dim, z_dim))
        
decoder = nn.Sequential(
            nn.Linear(z_dim, im_size*n_blocks*n_blocks),
            ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], ks=1, act_cls=nn.ReLU, norm_type=None),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=nn.ReLU, norm_type=None),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(nfs[2], 3, 3, padding=1),
            #nn.Dropout2d(dropout),
            nn.Sigmoid()
        )
#last_size

#             nn.Linear(latent_dim,hidden_dim), #nn.Linear(latent_dim, 16)
#             nn.Linear(hidden_dim,im_size*n_blocks*n_blocks), #nn.Linear(latent_dim, 16)
#             ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
#             ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[4], nfs[5], act_cls=Mish), #nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[5], nfs[6], act_cls=Mish), #nn.Conv2d(16, 1, 3),

            

inp=    torch.randn((32,3,160,160))

#encoder[:1](inp).shape
for ii in range(0,10):
    print(enc[:ii](inp).shape)

z = encoder(inp)
for ii in range(0,14):
    print(decoder[:ii](z).shape)

class UnFlatten(Module):
    def __init__(self, size=7):
        self.size = size
    def forward(self, input):
        return input.view(input.size(0), -1, self.size, self.size)

class MMD_VAE(Module):
    def __init__(self, latent_size):        
        self.encoder = nn.Sequential(
            ResBlock(1, 1, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 1, stride=2, act_cls=Mish),
            Flatten(),
            nn.Linear(16, latent_size)
        )
        
        dropout=0
        
        self.decoder = nn.Sequential(
            nn.Linear(latent_size, 16),
            UnFlatten(4),
            ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((28, 28)),
            nn.Sigmoid()
        )
        
    
    def forward(self, X):
        latent = self.encoder(X)
        return self.decoder(latent), latent

    
#decoder
n_blocks = 5        
nfs = [3] + [2**i*n_blocks for i in range(n_blocks+1)] 
nfs.reverse()
n = len(nfs)


tst = nn.Sequential(
            nn.Linear(latent_dim,hidden_dim, #nn.Linear(latent_dim, 16)
            nn.Linear(hidden_dim,im_size*n_blocks*n_blocks) #nn.Linear(latent_dim, 16)
            UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((3,im_size, im_size)),
            SigmoidRange(*out_range)#nn.Sigmoid()

    
                                         *modules,
                                      ConvLayer(nfs[-2],nfs[-1],
                                                ks=1,padding=0, norm_type=None, #act_cls=nn.Sigmoid) )
                                                act_cls=partial(SigmoidRange, *out_range)))

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

n_epoch = 10
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

n_epoch = 40
#learn.unfreeze()
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.05)

#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"frozen{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

x=1
x

def create_encoder(nfs,ks,conv=nn.Conv2d,bn=nn.BatchNorm2d,act_fn = nn.ReLU):
    """
    constructor for generic convolutional encoder 
    """
    n = len(nfs)
    conv_layers = [nn.Sequential(ConvBnRelu(nfs[i],nfs[i+1],kernel_size=ks[i],
                                            conv = conv,bn=bn,act_fn=act_fn, padding = ks[i] //2 ),
                                 Downsample(channels=nfs[i+1],filt_size=3,stride=2))
                                   for i in range(n-1)]        
    convs = nn.Sequential(*conv_layers)
    return convs

def create_encoder_denseblock(n_dense,c_start):
    """
    constructor for resnet with dense blocks  (?) 

    n_dense": 3,
    "c_start": 4
    """
    first_layer = nn.Sequential(ConvBnRelu(3,c_start,kernel_size=3,padding = 1),
                                ResBlock(c_start),
                                Downsample(channels=4,filt_size=3,stride=2))
    
    layers = [first_layer] + [
        nn.Sequential(
            DenseBlock(c_start * (2**c)),
            Downsample(channels=c_start * (2**(c+1)),filt_size=3,stride=2)) for c in range(n_dense)
    ]
    
    model = nn.Sequential(*layers)
    
    return model

def create_decoder(nfs, ks, size, conv=nn.Conv2d, bn=nn.BatchNorm2d, act_fn=nn.ReLU):
    """
    CURR VALUES:
    "nfs":[66,3*32,3*16,3*8,3*4,3*2,3,1,3],
    "ks": [ 3, 1, 3,1,3,1,3,1],   
    "size": IMG_SIZE 
    """
    n = len(nfs)
    
    # We add two channels to the first layer to include x and y channels
    first_layer = ConvBnRelu(nfs[0], #input size 
                             nfs[1], # output size
                             conv = PointwiseConv,
                             bn=bn,
                             act_fn=act_fn)

    conv_layers = [first_layer] + [ConvBnRelu(nfs[i],nfs[i+1],kernel_size=ks[i-1],
                                              padding = ks[i-1] // 2,conv = conv,bn=bn,act_fn=act_fn)
                                   for i in range(1,n - 1)]        
    dec_convs = nn.Sequential(*conv_layers)
    
    dec = nn.Sequential(SpatialDecoder2D(size),dec_convs)
    #SigmoidRange(*y_range)
    return dec

def decoder_simple(y_range=OUT_RANGE, n_out=3):
    return nn.Sequential(#UpsampleBlock(64),
                         UpsampleBlock(32),
                         nn.Conv2d(16, n_out, 1),
                         SigmoidRange(*y_range)
                        )

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	61706.023438	59401.890625	59200.527344	-0.183162	0.670351	-1.360018	1074.730103	0.182537	64435.050781	1.328313	0.744954	1.561802	00:22
1	58114.496094	54704.398438	54106.003906	-0.067606	0.709372	-1.241548	981.183960	0.604076	61621.859375	1.449838	0.450837	1.893286	00:22
2	55027.734375	52449.945312	52048.351562	-0.016363	0.778369	-0.767656	417.512909	0.947244	60466.074219	0.887459	0.376451	1.181295	00:22
3	53234.660156	51567.859375	51215.320312	-0.014475	0.771869	-0.710858	346.533447	1.000000	59968.601562	0.823350	0.310374	1.014969	00:22
4	52392.648438	51421.828125	51092.546875	-0.048797	0.782629	-0.654477	323.230713	1.000000	59896.394531	0.809924	0.289300	0.944704	00:22

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	73116.570312	56011.906250	41707.714844	-0.162279	0.150928	-43.849857	14298.664062	1.000000	53618.125000	1.220086	0.524610	36.704914	00:22
1	55711.816406	45964.726562	33920.769531	-0.043475	0.279428	-35.963737	12039.008789	1.000000	47239.265625	1.312494	0.757988	37.766373	00:22
2	44836.546875	37305.648438	27406.685547	-0.023731	0.321513	-29.381834	9895.095703	1.000000	41459.550781	1.460417	0.538946	35.474781	00:22
3	36200.839844	30034.455078	21493.714844	-0.048967	0.376191	-24.223383	8544.314453	1.000000	35556.191406	1.591922	0.687644	33.419842	00:22
4	29243.531250	26322.974609	17289.226562	-0.076567	0.462568	-20.003771	9098.589844	1.000000	30504.798828	1.651102	1.249426	31.537554	00:23
5	23948.765625	20693.746094	14159.992188	0.015880	0.476163	-17.972431	6532.034668	1.000000	27060.267578	1.679238	0.613610	30.386091	00:22
6	18914.341797	16118.884766	10100.545898	0.071780	0.485414	-15.699727	6016.059082	1.000000	21769.064453	1.614907	0.813107	28.720329	00:22
7	16302.161133	16133.471680	9065.597656	-0.032543	0.528036	-14.508196	7064.763672	1.000000	20103.945312	1.672818	0.956630	27.728195	00:23
8	14035.663086	13486.658203	8366.876953	-0.014977	0.549360	-13.761081	5118.403809	1.000000	18096.849609	1.596557	0.572698	27.181425	00:23
9	12555.615234	12438.543945	7636.028809	-0.001427	0.551520	-12.375941	4801.065430	1.000000	16884.033203	1.493703	0.722778	25.457499	00:22
10	11842.099609	11512.832031	7183.715820	0.014809	0.537542	-11.792240	4328.155762	1.000000	15679.622070	1.466000	0.521400	24.649998	00:22
11	10849.121094	10924.213867	6975.569824	-0.034562	0.546702	-10.934856	3947.825195	1.000000	15087.824219	1.373056	0.457514	23.260689	00:22
12	10149.152344	10338.746094	6664.760742	0.058513	0.561050	-10.111523	3673.358154	1.000000	14550.804688	1.332573	0.493117	22.023117	00:22
13	9612.157227	9841.376953	6494.279785	-0.028188	0.523342	-9.359609	3346.479980	1.000000	14008.334961	1.275519	0.430882	20.670998	00:22
14	9164.250977	9458.142578	6349.508789	0.017441	0.548174	-8.786793	3108.029785	1.000000	13718.131836	1.209039	0.398604	19.754648	00:22
15	8761.227539	9177.581055	6264.538574	0.044239	0.542927	-8.290083	2912.659668	1.000000	13411.566406	1.172588	0.369216	18.736010	00:22
16	8444.499023	8896.747070	6152.080078	-0.001141	0.557176	-7.828480	2744.129395	1.000000	13307.772461	1.127717	0.398963	17.916582	00:22
17	8178.131836	8776.557617	6145.216309	0.005876	0.548124	-7.555161	2630.786133	1.000000	13287.386719	1.102357	0.376227	17.280994	00:22
18	8179.623535	8713.017578	6129.683105	0.007566	0.553797	-7.419802	2582.790527	1.000000	13186.386719	1.086193	0.387691	17.034954	00:22
19	7979.851562	8669.065430	6088.716309	0.003607	0.549836	-7.418787	2579.850342	1.000000	13166.111328	1.084882	0.384289	17.028463	00:23

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	35639.734375	33119.378906	32758.919922	354.753448	-0.645665	2.168537	1.389641	45340.976562	5.047215	5219.156250	18799.441406	0.807266	00:24
1	22419.580078	15466.881836	15300.407227	162.598511	-0.390403	2.085497	1.324767	30500.722656	3.677496	3760.603516	10589.691406	0.736531	00:23
2	14881.570312	10914.019531	10817.428711	94.298996	-0.218614	1.718532	0.952760	24908.886719	2.857116	2908.460693	6201.050781	0.693021	00:23
3	10775.079102	8509.122070	8455.727539	51.379539	-0.116926	1.412444	0.578432	21353.609375	2.194111	2228.759521	3532.319580	0.640258	00:24
4	8417.246094	7147.596680	7119.229980	26.634216	-0.072432	1.150160	0.179836	18936.121094	1.669925	1692.649536	1983.132935	0.602620	00:24
5	6787.373047	6221.455566	6203.451660	16.419044	-0.051861	0.946574	-0.200824	17108.943359	1.339317	1349.057983	1275.688110	0.569902	00:23
6	5708.291504	5401.690430	5387.585449	12.589787	-0.049178	0.882773	-0.334380	15516.946289	1.213070	1216.506592	1068.951294	0.555498	00:23
7	4967.996582	4810.905273	4799.205566	10.367170	-0.035553	0.831857	-0.451990	14039.386719	1.123102	1120.465332	946.840881	0.552006	00:24
8	4480.203613	4512.324219	4501.353516	9.820558	-0.032393	0.812645	-0.496762	13347.663086	1.094568	1089.507446	911.750488	0.546974	00:23
9	4122.484863	4195.836426	4185.203613	9.572933	-0.030994	0.816746	-0.486494	12404.958984	1.091210	1083.963379	904.418335	0.547605	00:24
10	3843.719238	3955.936768	3945.632324	9.421382	-0.027061	0.806554	-0.510669	11746.567383	1.079526	1070.157715	891.319214	0.544316	00:24
11	3627.255371	3750.461670	3739.786133	9.974444	-0.035787	0.810802	-0.502045	11090.406250	1.095940	1082.943848	915.384888	0.550083	00:24
12	3453.144775	3555.236572	3544.947998	9.497383	-0.033756	0.805314	-0.512734	10532.422852	1.081325	1065.357910	893.720581	0.540843	00:24
13	3317.225586	3445.394287	3435.099854	9.603128	-0.026698	0.807787	-0.506707	10088.870117	1.086125	1070.625977	898.874268	0.540827	00:23
14	3207.483887	3477.685303	3467.043945	9.804630	-0.036560	0.803424	-0.516729	10080.664062	1.088416	1068.231079	904.268982	0.537274	00:23
15	3097.683838	3296.834473	3286.019287	10.156882	-0.024965	0.809974	-0.501416	9402.141602	1.100944	1077.333496	919.126099	0.541100	00:24
16	3026.498535	3239.862305	3229.419434	9.751334	-0.027817	0.814940	-0.486714	9327.415039	1.095807	1070.757568	907.116394	0.533075	00:23
17	2942.689209	3145.737793	3134.896729	10.162513	-0.031712	0.819668	-0.476477	8857.916992	1.109087	1079.275024	925.773315	0.536413	00:24
18	2910.235596	3092.333984	3081.208252	10.555488	-0.016682	0.829229	-0.454906	8585.517578	1.126567	1096.386475	948.013977	0.542863	00:23
19	2832.958252	3017.066650	3006.289307	10.245089	-0.018359	0.826162	-0.462675	8570.823242	1.114929	1082.901733	933.252991	0.543466	00:23
20	2772.829590	3046.259277	3036.564209	9.047776	-0.024695	0.789895	-0.547899	8529.547852	1.059283	1029.493530	866.595886	0.529360	00:24
21	2753.147461	2986.807861	2975.513428	10.669466	-0.013018	0.844861	-0.416567	8073.350098	1.141360	1104.890015	962.535461	0.538116	00:23
22	2712.455566	2906.891602	2896.313965	10.020215	-0.016617	0.819964	-0.475952	8024.366211	1.107157	1071.144531	922.119934	0.536537	00:23
23	2677.711914	3020.316895	3009.638916	10.101330	-0.021751	0.810849	-0.496884	8298.047852	1.099663	1062.236694	914.141846	0.532246	00:23
24	2626.546875	2862.383057	2852.536377	9.309927	-0.015195	0.803757	-0.513902	8120.997070	1.077071	1039.144897	884.831543	0.530815	00:23
25	2575.743164	2870.408691	2860.929199	9.051567	-0.026579	0.797159	-0.528016	8179.134766	1.062291	1025.824219	865.369934	0.521008	00:23
26	2480.682617	2658.409668	2648.303711	9.851988	-0.019407	0.812116	-0.495792	7458.644043	1.096348	1055.604858	911.445984	0.536942	00:24
27	2367.523193	2564.386475	2555.031738	9.078519	-0.010155	0.785778	-0.558931	7359.857910	1.055575	1014.959229	864.135681	0.527093	00:23
28	2287.362793	2577.752686	2568.370605	8.995213	-0.011609	0.781813	-0.570015	7215.798340	1.051848	1011.241882	862.347351	0.529978	00:24
29	2229.250488	2449.017822	2440.301514	8.443208	-0.008403	0.766377	-0.607935	7179.539551	1.025925	984.454834	834.755188	0.523161	00:23
30	2194.789062	2457.297119	2448.329590	8.684980	-0.008512	0.759338	-0.627476	7038.735840	1.025120	981.841309	838.818848	0.527536	00:24
31	2152.750732	2365.702148	2356.650146	8.731876	-0.015857	0.768805	-0.602370	6879.909668	1.036128	993.249329	847.809448	0.525540	00:23
32	2104.733643	2339.498535	2330.842041	8.281164	-0.006147	0.755411	-0.635174	6761.827637	1.013497	971.661621	823.330444	0.517066	00:24
33	2079.943604	2352.695068	2343.459961	8.989347	0.000814	0.767590	-0.606251	6705.492676	1.039513	994.690674	854.078735	0.525999	00:23
34	2057.652100	2289.373535	2281.190186	8.049372	-0.003593	0.754654	-0.636665	6767.381348	1.005222	961.257935	812.971985	0.514499	00:23
35	2048.321533	2333.577881	2325.027100	8.287884	-0.004861	0.745774	-0.659917	6811.984375	1.004853	959.555969	817.796509	0.513304	00:23
36	2008.904785	2287.277344	2278.947998	8.144099	-0.007205	0.744801	-0.662207	6504.483887	1.000480	952.984863	812.662354	0.512287	00:24
37	1978.226929	2199.820557	2191.496582	8.082468	-0.014684	0.740286	-0.675295	6599.623535	0.994912	947.844360	809.869019	0.514765	00:24
38	1965.509399	2354.071045	2345.678467	8.060910	-0.011115	0.751032	-0.645638	6712.160156	1.002705	954.138611	811.483521	0.512083	00:23
39	1944.204224	2543.402832	2531.188232	11.698755	-0.002674	0.802894	-0.519100	6909.925781	1.131840	1088.012329	967.922241	0.531437	00:23
40	1927.125122	2166.637451	2157.772949	8.694813	0.004934	0.754433	-0.638231	6360.273926	1.020341	970.247681	834.338135	0.515969	00:24
41	1904.941650	2148.423584	2138.998535	9.175574	-0.004919	0.763749	-0.616479	6247.569824	1.041988	990.484131	860.605591	0.524700	00:23
42	1880.683228	2143.890137	2136.084473	7.667233	-0.005195	0.724807	-0.717712	6384.495117	0.971001	920.460571	791.137024	0.514054	00:23
43	1868.364868	2138.596680	2130.624268	7.770609	-0.003167	0.732386	-0.696039	6208.037109	0.981391	929.703735	797.666382	0.510722	00:23
44	1873.806519	2115.231445	2106.587646	8.408823	0.002722	0.759259	-0.626507	6110.434082	1.019319	965.254517	830.953125	0.517964	00:23
45	1857.733521	2118.644287	2109.715576	8.607017	-0.003585	0.758680	-0.628329	6144.351074	1.024203	970.119385	838.559998	0.518992	00:24
46	1835.785889	2301.718750	2293.943604	7.280443	0.002950	0.731905	-0.693791	6752.790039	0.966830	915.910034	775.821228	0.498173	00:24
47	1826.181519	2070.978271	2062.959473	7.751598	-0.006774	0.736675	-0.683394	6289.015137	0.984957	930.847839	798.094299	0.506730	00:24
48	1816.817261	2169.010010	2158.630859	10.077996	0.000032	0.782318	-0.571635	6146.196777	1.079840	1023.483887	905.510681	0.533880	00:23
49	1790.655762	2093.010254	2083.828613	8.849428	-0.004012	0.751683	-0.650315	6050.121094	1.025513	967.276062	848.441895	0.530520	00:23
50	1793.118286	2089.919434	2081.775391	7.915750	-0.009043	0.746823	-0.658815	6050.777344	0.996809	939.305908	809.562744	0.515972	00:24
51	1784.936768	1993.558228	1985.290527	8.115150	0.003885	0.743259	-0.670213	5850.540527	0.999537	940.279785	817.424438	0.521515	00:23
52	1767.381958	2151.812012	2144.149658	7.295479	-0.002681	0.714391	-0.744198	6923.266113	0.954031	902.872864	775.648926	0.504909	00:24
53	1756.909668	2133.470947	2125.688232	7.532532	0.006581	0.730068	-0.702234	6150.624512	0.973271	915.361450	789.177856	0.508780	00:23
54	1746.442627	2092.561768	2084.928467	7.357768	-0.000443	0.728496	-0.706767	6296.643555	0.968090	909.576660	784.124512	0.510359	00:24
55	1744.525879	2060.050049	2051.518799	8.250493	-0.002866	0.753495	-0.642588	5905.169922	1.010369	948.708313	824.126892	0.520684	00:24
56	1725.987915	2222.684082	2211.968994	10.327498	0.001512	0.780892	-0.574243	6259.079590	1.084146	1023.592651	910.862427	0.530152	00:24
57	1713.938354	2353.004395	2345.289307	7.142781	-0.002146	0.695641	-0.793052	8155.063477	0.933386	889.344971	761.682495	0.489615	00:23
58	1705.436157	1990.490601	1982.330688	7.960275	0.006327	0.741737	-0.672134	5864.037109	0.992518	932.534546	807.071838	0.514355	00:24
59	1705.450562	2019.981323	2010.744507	8.982873	-0.000409	0.759453	-0.628868	5937.989746	1.035373	972.338074	855.795715	0.526218	00:24
60	1696.698486	1959.749512	1951.297974	8.261941	-0.000039	0.743128	-0.669158	5735.625488	1.001879	939.263184	818.803406	0.516575	00:24
61	1696.808472	2162.159180	2151.416260	10.234275	0.011562	0.773763	-0.594086	6051.969727	1.078619	1015.008423	909.483826	0.533412	00:24
62	1681.989136	1939.605103	1931.698853	7.813064	-0.000969	0.728840	-0.709018	5738.298340	0.978495	914.541321	800.384216	0.520435	00:24
63	1667.958008	1908.849731	1901.618042	6.975263	0.005093	0.717563	-0.736385	5811.500488	0.948977	888.578186	768.285400	0.507422	00:24
64	1657.080566	1997.998535	1990.892212	6.848620	0.003718	0.708017	-0.761569	6328.044922	0.936035	878.318420	758.598022	0.501764	00:23
65	1650.266357	1880.462524	1872.814941	7.524014	-0.000007	0.728828	-0.709459	5613.864746	0.972260	908.505615	793.400818	0.520965	00:23
66	1644.701172	1923.961914	1916.933350	6.810940	0.001214	0.712433	-0.750617	5899.688477	0.939393	879.358093	760.829346	0.506575	00:24
67	1660.577271	1908.370361	1901.273438	6.838502	-0.000657	0.715031	-0.742947	5754.287109	0.942613	881.583496	762.272522	0.505253	00:23
68	1642.703491	2075.767090	2068.711670	6.701159	0.004295	0.686344	-0.822066	6988.598633	0.914129	860.682068	749.578125	0.495988	00:23
69	1637.802124	1936.513306	1927.819946	8.439760	0.001987	0.747285	-0.662898	5694.429199	1.013492	942.569397	836.994812	0.531298	00:24
70	1628.861938	1855.664185	1848.401855	7.107107	0.004089	0.721452	-0.729289	5562.158203	0.956804	892.012207	779.056335	0.518865	00:23
71	1620.777588	1849.363525	1841.987549	7.230100	0.005408	0.727281	-0.712910	5576.687988	0.964420	897.751221	783.769714	0.518859	00:23
72	1617.906738	1872.577148	1865.401001	7.024238	0.001375	0.722295	-0.724200	5726.461914	0.953735	888.874390	771.711609	0.510274	00:23
73	1605.725342	1879.347168	1872.517212	6.652784	0.001630	0.705851	-0.771451	5707.089355	0.931615	866.731323	759.250916	0.513576	00:24
74	1601.359619	1860.889526	1853.274292	7.449453	0.000522	0.732364	-0.699802	5518.531738	0.974449	906.497131	793.526978	0.520268	00:23
75	1593.129028	1819.535645	1812.225342	7.186706	-0.003417	0.723173	-0.724752	5449.743164	0.959148	890.988892	781.059082	0.519311	00:24
76	1598.570190	1943.324585	1936.156006	6.982880	0.001308	0.717146	-0.740403	5724.291016	0.948234	881.817749	770.956482	0.515682	00:23
77	1596.463501	1858.350708	1851.585205	6.690332	0.002513	0.700747	-0.784782	5736.254395	0.925605	861.278259	755.052063	0.510597	00:24
78	1591.501099	2161.671875	2154.538330	6.567418	-0.001495	0.686647	-0.821709	7697.672363	0.912659	860.494751	748.370300	0.496979	00:24
79	1596.576050	1848.325806	1841.201050	6.923673	-0.007202	0.719312	-0.734420	5649.215820	0.949537	881.026062	771.243774	0.515774	00:23
80	1578.273560	1850.652710	1843.813721	6.662688	-0.002183	0.708437	-0.763810	5656.493652	0.932729	867.856689	758.068176	0.512917	00:23
81	1558.910889	1857.016357	1850.471191	6.282830	0.003750	0.694354	-0.801634	6025.386719	0.911618	850.710693	742.491760	0.504756	00:23
82	1566.326172	1877.583496	1870.850586	6.587173	-0.000365	0.712432	-0.751816	5813.219238	0.933901	868.544617	755.775269	0.509591	00:24
83	1558.945923	1796.186035	1788.957031	7.010788	0.008110	0.711451	-0.757494	5445.705566	0.945560	876.531738	773.538818	0.519563	00:24
84	1563.117065	2025.193726	2015.877319	8.986026	0.001242	0.749838	-0.658024	5764.464355	1.027073	952.846375	855.554688	0.537086	00:24
85	1562.327148	1820.796997	1813.988525	6.611163	0.000196	0.713180	-0.751190	5506.539551	0.936482	867.845459	759.705750	0.513956	00:23
86	1544.917236	1775.630371	1768.784912	6.705034	0.005688	0.707357	-0.769205	5401.282227	0.933454	863.741150	762.567932	0.519356	00:24
87	1546.165283	1824.941895	1817.632690	7.065378	0.001202	0.721854	-0.730349	5397.050293	0.955980	882.884644	780.678223	0.524164	00:23
88	1546.792969	1851.563965	1844.506348	6.900307	0.006732	0.710861	-0.760693	5496.184570	0.942076	870.337891	771.701538	0.523635	00:24
89	1533.520386	1848.635742	1840.505615	8.000067	0.002263	0.733689	-0.702156	5597.409668	0.991777	915.240845	821.559875	0.538800	00:24
90	1533.974854	2042.176758	2033.418335	8.438148	-0.000659	0.734213	-0.701345	5859.502441	1.004081	928.179810	837.471558	0.540214	00:24
91	1534.598755	1770.096680	1762.933228	6.965265	0.000781	0.721685	-0.730158	5302.961426	0.953490	880.158203	776.901062	0.521827	00:24
92	1529.138550	1769.280640	1762.805542	6.223247	0.000261	0.700388	-0.789683	5462.433594	0.917643	848.088928	750.385254	0.521262	00:23
93	1514.251465	1756.898438	1750.141479	6.658386	-0.001312	0.703789	-0.780196	5348.198242	0.930945	860.246765	763.566895	0.522191	00:24
94	1511.639648	1742.198242	1735.486206	6.530146	0.004555	0.707732	-0.769576	5264.849121	0.932178	860.377258	762.482971	0.523589	00:24
95	1518.488770	1809.066895	1802.725586	6.125498	0.004147	0.690996	-0.813873	5853.875488	0.904323	839.218445	739.848206	0.511833	00:24
96	1506.064941	1757.221313	1750.776855	6.236708	-0.000482	0.703213	-0.782113	5332.096680	0.919954	849.483337	751.440063	0.521793	00:23
97	1492.611450	1749.808716	1743.591553	6.031801	0.008199	0.693211	-0.808296	5335.791016	0.903719	836.115479	738.152649	0.514672	00:24
98	1495.907227	1780.331055	1774.109863	5.997238	0.006235	0.682826	-0.837528	5565.546875	0.893420	827.784546	734.499817	0.511417	00:23
99	1506.924927	1753.827026	1747.724487	5.929687	0.008335	0.687965	-0.823548	5548.875977	0.897357	830.657043	735.534119	0.514324	00:24
100	1490.491211	1772.699097	1765.414429	7.126364	0.005110	0.712210	-0.759742	5342.322754	0.951754	876.143677	785.646484	0.531999	00:24
101	1490.735229	2094.084229	2087.595459	6.007836	0.017847	0.672853	-0.859687	7370.563965	0.883705	832.604309	725.148865	0.486253	00:24
102	1474.431885	1886.980835	1880.927002	5.964793	0.002663	0.685117	-0.831594	5843.044922	0.893664	828.043518	733.632629	0.513065	00:23
103	1488.557861	1763.483521	1757.162354	6.224775	0.003430	0.689050	-0.823036	5342.456055	0.904692	833.983154	745.886536	0.522442	00:23
104	1478.421631	1721.488403	1714.755249	6.576198	0.002086	0.705786	-0.775474	5198.592285	0.929646	855.405090	761.574036	0.523895	00:24
105	1477.666626	1734.629395	1728.487549	5.992128	-0.000108	0.690119	-0.818457	5326.170410	0.900211	829.373230	738.171143	0.517729	00:24
106	1473.125854	1963.250122	1954.735840	8.192457	-0.009232	0.725935	-0.725695	5698.272461	0.992598	912.751648	830.933044	0.544388	00:23
107	1470.645142	1754.682129	1748.505615	6.028650	-0.003051	0.689379	-0.821751	5312.978516	0.901407	830.095337	741.401245	0.522096	00:23
108	1466.332764	1757.086548	1750.146240	6.846844	-0.000674	0.705047	-0.780684	5243.905273	0.935956	859.592285	773.074219	0.533158	00:23
109	1469.749634	1726.563477	1720.450806	5.911416	0.003956	0.687264	-0.826549	5249.537598	0.896469	825.995483	736.348145	0.517084	00:23
110	1464.484375	1736.870850	1730.732300	5.962198	0.004571	0.686533	-0.828575	5334.952637	0.897237	826.512878	737.748352	0.516293	00:24
111	1474.863647	1793.794189	1788.000122	5.671723	-0.001588	0.674181	-0.861681	5883.505371	0.876902	814.074036	722.494751	0.505507	00:24
112	1462.521729	1711.135254	1704.977295	6.024343	0.000763	0.687511	-0.827819	5293.791016	0.900351	829.086060	742.556763	0.523178	00:24
113	1449.825928	1701.789673	1695.721924	5.878519	0.001709	0.687487	-0.828042	5216.040527	0.896621	825.075745	738.509460	0.524055	00:24
114	1448.282471	1792.912231	1785.387207	7.269108	0.003752	0.716244	-0.750997	5364.237305	0.961296	881.468506	797.154053	0.539143	00:24
115	1445.574585	1739.160400	1733.344604	5.582691	0.000341	0.675874	-0.857014	5535.911621	0.877061	811.681335	721.563904	0.506585	00:24
116	1447.305664	1702.951294	1696.921875	5.905175	-0.001069	0.685150	-0.835695	5261.783203	0.895340	821.455139	740.116638	0.526518	00:24
117	1440.761230	1798.639282	1791.698853	6.732737	0.006422	0.698484	-0.798006	5357.394531	0.930286	853.307495	770.295471	0.528635	00:24
118	1436.066528	1683.260254	1676.651978	6.451030	0.004302	0.699094	-0.796131	5083.051270	0.921508	844.567322	759.091492	0.528947	00:24
119	1435.049683	1834.709106	1827.218750	7.215334	0.002182	0.712799	-0.759363	5444.700195	0.956702	878.528870	792.816406	0.534509	00:23
120	1432.722534	1712.610962	1706.566895	5.798853	-0.005328	0.687854	-0.825481	5167.812500	0.894212	821.314392	734.109131	0.517869	00:23
121	1434.100586	1951.651611	1945.811279	5.471668	0.007534	0.673743	-0.859499	7142.018066	0.870716	811.510376	712.458984	0.492756	00:23
122	1437.508789	1674.697021	1668.088135	6.481055	-0.004835	0.701548	-0.789016	5056.125000	0.924078	847.228516	760.208435	0.528015	00:24
123	1434.666138	1735.760132	1730.124634	5.366279	0.007576	0.676784	-0.854084	5562.921875	0.872821	807.179810	715.587524	0.505585	00:24
124	1429.497192	1689.528076	1683.750244	5.527700	0.002926	0.677468	-0.853770	5230.196777	0.877476	808.014038	722.075500	0.511612	00:24
125	1418.321533	2098.251709	2087.811279	9.920640	0.010520	0.737638	-0.696218	5841.585449	1.045698	967.533997	895.270081	0.550583	00:23
126	1417.721924	1687.144531	1680.525635	6.484304	0.001014	0.700080	-0.794815	5142.148926	0.925255	845.612244	764.709595	0.532689	00:24
127	1415.999878	1661.122559	1654.787842	6.252984	-0.001776	0.690348	-0.821688	5054.491211	0.908621	830.860168	752.067505	0.529559	00:24
128	1419.004517	1668.103516	1661.842285	6.131657	0.004399	0.683106	-0.842830	5066.711914	0.900992	823.832336	749.567566	0.530140	00:24
129	1428.106201	1646.717651	1640.340942	6.264211	0.003743	0.692410	-0.817175	5054.361328	0.912708	833.564209	756.597290	0.533775	00:24
130	1412.550659	1658.395264	1652.331421	5.964541	0.005635	0.685173	-0.836093	5059.983887	0.897434	821.432312	743.021545	0.527028	00:24
131	1400.915649	1694.892700	1689.228394	5.458885	0.000171	0.670887	-0.872974	5170.104980	0.869605	799.230408	719.464050	0.509511	00:23
132	1415.313843	1657.355225	1650.954102	6.333950	-0.001637	0.689140	-0.826907	5043.656738	0.910786	832.046265	757.605103	0.534587	00:23
133	1410.529419	1861.309570	1853.625488	7.459136	0.000599	0.704333	-0.785786	5443.997070	0.955456	874.413452	800.895813	0.541768	00:24
134	1403.009766	1903.692871	1898.027344	5.320584	0.005749	0.659000	-0.902107	6834.922852	0.851381	795.139038	703.956909	0.486862	00:23
135	1399.830078	1701.127319	1695.625732	5.258708	0.001896	0.668509	-0.880920	5312.633301	0.862366	793.509644	714.375000	0.512364	00:24
136	1398.415649	1738.159668	1732.829834	5.186443	0.006218	0.673360	-0.863564	5501.898926	0.863448	798.265930	707.888245	0.502401	00:24
137	1399.475586	1891.271118	1885.845581	5.066854	0.005012	0.655829	-0.913119	6470.983398	0.843407	784.407654	700.131470	0.491701	00:24
138	1406.035889	1675.015747	1669.492188	5.342115	0.003427	0.678883	-0.849478	5177.966797	0.872906	802.791809	715.483704	0.510385	00:23
139	1396.128906	1638.963135	1633.346924	5.520921	0.002290	0.671443	-0.874290	5054.814453	0.871630	797.175049	724.165344	0.518922	00:24
140	1389.001709	1951.423584	1945.744263	5.251068	0.002636	0.661194	-0.898086	6788.769043	0.854937	794.803528	708.418152	0.495433	00:24
141	1388.087158	1688.355103	1682.019043	6.212550	-0.000564	0.681996	-0.846479	5130.877441	0.901388	823.665833	751.761047	0.530002	00:23
142	1385.485352	1639.908691	1633.910156	5.914208	0.000989	0.685338	-0.834779	5048.670410	0.895893	819.125366	740.246887	0.524021	00:24
143	1389.624268	1728.018799	1722.430908	5.454906	-0.002389	0.675913	-0.859976	5214.032715	0.873538	800.997437	720.956177	0.515619	00:23
144	1387.876587	1627.204834	1621.583496	5.461523	0.002939	0.666704	-0.888658	5018.274414	0.867548	794.538818	724.534485	0.519912	00:24
145	1395.786377	1799.681152	1794.277466	5.198103	0.004318	0.672103	-0.868977	5674.524414	0.863514	795.230957	710.991943	0.507598	00:23
146	1389.711426	2341.362305	2330.410645	10.206139	0.009791	0.737460	-0.699181	6225.690918	1.055980	972.888184	912.047424	0.557112	00:24
147	1382.315186	1889.569336	1882.606689	6.672486	0.002923	0.696264	-0.806550	5563.912109	0.928896	847.870911	773.047607	0.534387	00:23
148	1381.061890	1683.992554	1678.705200	5.088257	-0.000407	0.666899	-0.882866	5115.092285	0.854922	787.304565	704.907288	0.501818	00:24
149	1381.243164	1756.639404	1751.480225	5.025988	-0.004411	0.664984	-0.886566	5821.260254	0.849539	785.729736	698.703003	0.494932	00:24
150	1391.396118	1629.302002	1623.264282	5.902591	0.003420	0.686533	-0.832479	4975.734863	0.897541	818.911499	742.386169	0.527404	00:23
151	1380.467407	1841.547485	1836.219360	4.929632	0.006568	0.659674	-0.899792	6095.972168	0.843260	786.319092	694.334778	0.485277	00:23
152	1376.042358	1656.519165	1650.619995	5.746026	-0.001170	0.687702	-0.827185	5076.075684	0.892985	817.150696	734.218079	0.520622	00:24
153	1368.910645	1604.249878	1598.579346	5.557160	0.005134	0.675994	-0.862590	4929.545898	0.878161	801.331299	729.264526	0.524311	00:24
154	1364.348389	1711.368652	1705.100952	6.029387	0.007552	0.679768	-0.852199	5120.926270	0.894755	816.590027	745.362122	0.526926	00:24
155	1364.940796	1623.401489	1618.064819	5.157538	0.002106	0.669850	-0.877013	5024.353027	0.860458	788.950806	711.364502	0.511635	00:23
156	1369.497925	1737.252075	1732.073120	5.001041	-0.000255	0.660906	-0.899281	5684.628906	0.846304	782.234314	699.924072	0.495823	00:23
157	1375.073242	1670.945557	1665.610962	5.146222	0.000239	0.675878	-0.859120	5231.878418	0.865907	794.046570	711.487976	0.511762	00:23
158	1361.945435	1659.720459	1653.842285	5.687163	0.004606	0.677694	-0.857958	5073.202637	0.884437	807.569641	735.100952	0.525402	00:24
159	1359.260254	1672.385132	1667.300415	4.952787	0.002986	0.662565	-0.895641	5178.009766	0.846214	779.041077	699.466980	0.500852	00:24
160	1349.149414	1640.890137	1634.643066	6.153680	0.003964	0.684842	-0.839263	5043.590820	0.904352	822.992798	753.826355	0.532841	00:23
161	1353.739624	1597.526001	1592.349854	5.021693	0.006057	0.660444	-0.905429	5003.083008	0.848760	777.452271	707.987366	0.511559	00:23
162	1347.540649	1638.532959	1633.320190	5.058012	0.000977	0.651229	-0.934468	5096.089844	0.841315	770.387146	710.256409	0.515415	00:24
163	1335.703491	1695.615967	1689.228760	6.245860	0.005231	0.682442	-0.846417	5214.112793	0.904372	825.082397	756.296143	0.532854	00:24
164	1346.623779	1590.747314	1585.051636	5.653391	0.000028	0.677425	-0.858526	4957.563965	0.882757	804.979065	733.336365	0.524897	00:24
165	1333.729492	1623.744019	1618.688232	4.874943	0.000402	0.646668	-0.947792	5077.589355	0.832516	762.127869	704.936768	0.512799	00:23
166	1333.856934	1700.821411	1695.920654	4.680223	0.000258	0.645109	-0.949244	5450.968750	0.823805	758.155823	694.016968	0.501195	00:24
167	1327.820557	1576.457520	1571.300537	5.078076	0.001820	0.654918	-0.923109	4919.488770	0.844439	772.418518	709.839233	0.514651	00:24
168	1328.926758	1573.758301	1568.447388	5.132746	0.002853	0.656624	-0.919550	4844.415527	0.850984	776.917114	716.910889	0.520079	00:24
169	1319.658813	1629.252441	1623.602295	5.531579	0.004089	0.663933	-0.900344	4970.062988	0.868671	791.313416	732.179138	0.529097	00:24
170	1311.670044	1567.386597	1562.363037	5.011216	-0.002171	0.646587	-0.949217	4992.060547	0.834543	762.800232	708.426880	0.516315	00:24
171	1311.979126	1565.485718	1560.121094	5.253644	0.003709	0.652157	-0.933970	4863.855469	0.849615	774.268250	720.804810	0.522821	00:24
172	1309.738159	1569.579224	1564.234375	5.176810	-0.001438	0.645019	-0.957222	4800.472168	0.841719	767.224060	721.150696	0.526419	00:24
173	1300.042969	1576.110474	1571.340088	4.693328	0.002788	0.635771	-0.981228	4908.370117	0.815928	747.155457	698.806824	0.510915	00:24
174	1300.506714	1596.154785	1591.422485	4.600417	0.000205	0.631884	-0.993400	5049.825195	0.811167	743.685730	698.102051	0.510750	00:24
175	1302.210083	1565.038940	1560.224731	4.715932	-0.002279	0.632513	-0.993281	4823.053223	0.815410	744.632507	703.829956	0.516547	00:24
176	1295.195312	1634.335571	1628.611816	5.639289	0.004621	0.643383	-0.965427	5045.840820	0.854679	777.336243	740.196472	0.536837	00:24
177	1296.105835	1552.511475	1547.794678	4.642930	-0.001359	0.625849	-1.015444	4811.358887	0.807235	736.801025	703.913940	0.519581	00:23
178	1289.839111	1555.052490	1550.240967	4.691702	0.002196	0.623276	-1.024005	4831.184082	0.807397	736.174866	707.429565	0.520973	00:23
179	1288.338867	1675.798340	1669.824585	5.750722	0.008604	0.646306	-0.958059	5084.871582	0.863302	785.137085	748.276550	0.542142	00:23
180	1283.673828	1516.768433	1511.867798	4.808212	0.000827	0.625714	-1.017968	4717.286621	0.813187	739.968689	712.312256	0.526445	00:24
181	1276.340210	1549.146240	1544.537231	4.559086	-0.000139	0.611956	-1.059541	4878.202148	0.791074	722.302429	703.156128	0.517453	00:24
182	1275.160889	1523.992432	1519.262573	4.653107	0.003621	0.616803	-1.046131	4775.168457	0.800713	729.311401	709.372437	0.525189	00:24
183	1274.201660	1535.204346	1530.678711	4.456763	0.004054	0.606965	-1.076548	4829.563477	0.784311	716.280945	703.056152	0.519575	00:23
184	1268.209961	1511.440186	1506.698242	4.603409	0.004325	0.610509	-1.067291	4754.063965	0.794557	723.420471	711.402893	0.527470	00:24
185	1266.569946	1543.884888	1539.392822	4.316476	0.002206	0.603420	-1.087745	4918.042969	0.778405	711.360107	701.144226	0.517730	00:24
186	1263.202515	1528.685181	1523.815796	4.723829	0.001786	0.614226	-1.057132	4734.013184	0.803246	729.756042	717.827942	0.533173	00:23
187	1260.967529	1508.121826	1503.532715	4.507143	0.000652	0.605073	-1.085063	4768.585449	0.786481	716.546814	709.896118	0.526683	00:24
188	1254.141235	1521.172729	1516.287964	4.772686	0.003552	0.610881	-1.069029	4713.490723	0.802140	728.285034	721.653503	0.536562	00:24
189	1258.827637	1500.663452	1496.237183	4.378232	0.001310	0.597949	-1.108888	4722.563965	0.776632	706.947815	709.265503	0.527378	00:24
190	1254.742188	1494.618530	1490.014893	4.517956	0.002078	0.599058	-1.107524	4685.872070	0.783946	711.739197	717.455383	0.534846	00:23
191	1253.978394	1573.777710	1569.478027	4.117863	0.000867	0.586330	-1.145558	5067.717773	0.757283	692.494629	703.100769	0.519190	00:23
192	1248.770020	1492.288940	1487.655640	4.550579	0.001854	0.598468	-1.109971	4670.904785	0.784323	711.917358	719.044434	0.536398	00:24
193	1247.504150	1494.558472	1489.838989	4.692032	0.002832	0.601183	-1.101408	4682.655273	0.789145	715.769653	720.914001	0.537973	00:24
194	1246.692139	1497.859863	1493.126465	4.654514	0.002181	0.602137	-1.098455	4664.011719	0.791181	717.602112	721.982056	0.538484	00:24
195	1248.151001	1487.668579	1483.009277	4.578956	0.002568	0.598033	-1.111644	4664.344238	0.783169	710.659485	718.671692	0.536958	00:24
196	1243.887207	1486.362183	1481.893433	4.425810	0.001585	0.593358	-1.126205	4675.741699	0.774591	703.536987	715.078613	0.533562	00:24
197	1238.384277	1485.278931	1480.723999	4.445567	0.001731	0.592773	-1.128412	4670.096680	0.774399	703.236633	715.891174	0.534319	00:24
198	1243.141846	1485.222656	1480.728638	4.453456	0.002115	0.593524	-1.125854	4668.957031	0.775106	703.962830	715.587891	0.534109	00:24
199	1240.411377	1485.554810	1481.032349	4.460090	0.001861	0.592433	-1.129834	4674.369629	0.774331	703.022034	716.537415	0.535203	00:24

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	44869.117188	38630.105469	38172.390625	452.347382	0.363450	2.003438	1.154238	49319.636719	4.148453	8080.807617	26520.181641	0.966020	00:23
1	28217.714844	19543.214844	19365.693359	173.174393	0.211772	1.590892	0.723334	34397.785156	2.722190	5338.421875	11395.580078	0.892636	00:23
2	18315.355469	13255.430664	13121.753906	130.631409	0.219888	1.532194	0.665691	27403.474609	2.465842	4917.343750	9348.974609	0.835872	00:23
3	12825.623047	9738.818359	9650.188477	86.012886	0.174977	1.367238	0.465474	22967.119141	2.081707	4158.813965	6535.408691	0.760577	00:24
4	9639.348633	7791.971191	7737.542480	52.770176	0.111142	1.187918	0.203920	20011.353516	1.711145	3402.236816	4313.964844	0.716786	00:24
5	7687.958496	6969.908203	6936.609375	31.604336	0.071600	1.017000	-0.096666	18651.474609	1.397718	2771.013672	2869.612793	0.689790	00:24
6	6420.611328	5956.696289	5929.208008	25.928036	0.055198	0.933785	-0.259582	16708.939453	1.273471	2517.689209	2402.669678	0.671029	00:24
7	5523.093750	5197.247559	5173.945801	21.902893	0.056021	0.890996	-0.346857	14908.278320	1.195464	2358.052979	2146.462646	0.652390	00:23
8	4918.751465	4908.517578	4887.394043	19.962851	0.048813	0.845242	-0.446389	13966.420898	1.135524	2239.118164	1974.152588	0.633212	00:24
9	4473.037598	4544.405762	4524.976074	18.338888	0.045379	0.825765	-0.490280	13264.149414	1.100679	2152.622803	1881.080322	0.627626	00:23
10	4142.837402	4177.356934	4157.032715	19.363203	0.038059	0.848681	-0.432753	12074.405273	1.130283	2209.923340	1944.031250	0.618873	00:24
11	3917.388916	3997.903320	3977.184814	19.701015	0.057482	0.848224	-0.430466	11531.174805	1.133727	2215.214111	1949.137207	0.608181	00:24
12	3699.811768	3793.642334	3773.311523	19.624884	0.058232	0.857292	-0.405696	10980.250000	1.137904	2227.622314	1948.683960	0.595239	00:24
13	3548.590576	3717.894043	3695.793945	21.325510	0.051739	0.856475	-0.406966	10586.673828	1.154981	2252.016113	1997.359985	0.593199	00:24
14	3392.223633	3564.068848	3543.844238	19.469976	0.048081	0.859627	-0.396350	10151.842773	1.136936	2212.456299	1934.764526	0.582036	00:24
15	3294.741943	3458.957520	3437.580811	20.652731	0.035036	0.864861	-0.385641	9724.040039	1.155857	2242.773926	1989.750732	0.587111	00:24
16	3188.714355	3471.833740	3450.045410	21.072681	0.061831	0.877749	-0.353019	9386.150391	1.169560	2276.136963	2020.442383	0.577378	00:24
17	3111.545898	3386.387207	3364.288574	21.481207	0.057307	0.876575	-0.355280	9266.531250	1.172527	2275.385742	2028.340576	0.576368	00:24
18	3067.808838	3286.744629	3263.543457	22.496641	0.058035	0.901485	-0.299061	8925.688477	1.206159	2334.083984	2121.989258	0.577001	00:23
19	3007.293945	3189.976807	3167.430908	21.941383	0.049392	0.884795	-0.334096	8704.542969	1.184379	2293.989990	2053.986328	0.570027	00:23
20	2945.777100	3182.058105	3159.409668	22.111326	0.049436	0.907009	-0.285508	8585.974609	1.205574	2330.225830	2113.958740	0.571989	00:23
21	2895.951416	3211.220459	3188.865967	21.689819	0.046961	0.886839	-0.327073	8588.288086	1.182075	2284.830322	2039.987183	0.562882	00:24
22	2849.264160	3051.117188	3028.877930	21.914141	0.062428	0.886953	-0.327097	8263.973633	1.183866	2291.922607	2047.876343	0.563708	00:24
23	2808.139648	3054.230713	3032.758301	20.943661	0.052907	0.880357	-0.340083	8256.331055	1.169980	2262.256104	2002.924072	0.556725	00:24
24	2773.731201	3043.178711	3019.842285	22.779768	0.056994	0.902545	-0.291591	8085.221680	1.206399	2338.058105	2110.115967	0.558648	00:23
25	2742.290527	2963.630859	2941.413818	21.570345	0.058292	0.876817	-0.347950	7992.745117	1.172185	2268.051758	2011.426392	0.555742	00:23
26	2719.287354	2948.922852	2925.617676	22.668314	0.061004	0.901012	-0.293574	7791.686523	1.204079	2328.265869	2100.627441	0.555158	00:23
27	2679.970215	2916.285645	2894.107422	21.886332	0.058098	0.897683	-0.300988	7793.752441	1.191654	2299.196289	2063.407471	0.554300	00:23
28	2657.288818	2888.732910	2867.198486	21.124582	0.054443	0.885014	-0.327630	7721.035645	1.174288	2261.996826	2010.303223	0.548887	00:23
29	2637.341553	2875.486816	2852.899658	22.185432	0.052402	0.899004	-0.297901	7635.975098	1.196017	2309.848877	2075.057373	0.552954	00:24
30	2611.630371	2916.487549	2893.916016	22.165520	0.060199	0.892029	-0.311832	7624.857910	1.190608	2294.882568	2058.446533	0.548067	00:23
31	2601.701660	2958.729736	2937.280029	21.088247	0.052859	0.875972	-0.348007	7780.184570	1.164899	2237.513916	1986.447510	0.547276	00:24
32	2560.690186	2890.773193	2868.005615	22.291035	0.053579	0.893000	-0.311326	7511.469238	1.193920	2301.701660	2071.535645	0.552681	00:24
33	2536.779785	2881.475586	2859.619873	21.348116	0.060162	0.896214	-0.303677	7471.465332	1.185748	2284.387695	2045.750488	0.550699	00:24
34	2520.879150	2919.730957	2896.383057	22.488171	0.056736	0.900711	-0.294199	7453.818848	1.202283	2313.757324	2095.693848	0.553407	00:23
35	2495.342773	2774.852539	2752.407227	21.960470	0.051357	0.895464	-0.304288	7264.982422	1.192231	2286.324463	2062.578125	0.546508	00:23
36	2484.418701	2731.136230	2710.085938	20.814085	0.050483	0.880643	-0.337369	7257.747559	1.166030	2236.843018	1987.916504	0.546926	00:23
37	2467.477051	2689.622559	2667.826416	21.342794	0.057822	0.896463	-0.301654	7114.496094	1.185970	2275.238037	2043.145142	0.544947	00:23
38	2457.845703	2991.763428	2970.408203	21.072306	0.052153	0.889041	-0.318461	7656.958008	1.176745	2257.115234	2017.733765	0.545500	00:24
39	2322.232178	2856.982422	2834.868896	21.692192	0.040446	0.881323	-0.335423	7360.048828	1.178361	2268.504395	2023.965210	0.540828	00:24
40	2227.645508	2579.064941	2557.928467	20.654335	0.044434	0.849737	-0.406745	6973.835449	1.140613	2198.822021	1924.729004	0.536686	00:24
41	2180.507080	2692.549805	2674.151611	18.129353	0.046520	0.833835	-0.442311	7177.255859	1.097045	2103.624512	1805.527344	0.531004	00:24
42	2126.195068	2674.731445	2655.945312	18.241838	0.050435	0.825340	-0.461466	7156.921875	1.091575	2089.953613	1794.163574	0.526470	00:23
43	2091.339111	2310.899902	2293.354248	17.364128	0.047284	0.810492	-0.496608	6644.475098	1.069493	2046.097046	1742.156250	0.522600	00:24
44	2053.124756	2347.489014	2329.062744	18.080427	0.043661	0.815187	-0.483483	6631.128418	1.079206	2075.353027	1761.062134	0.515697	00:24
45	2039.703735	2485.505127	2466.542236	18.650936	0.058622	0.830873	-0.447020	6744.937988	1.100019	2106.931396	1812.271118	0.521363	00:24
46	2014.683228	2239.049316	2220.724609	18.121702	0.051046	0.826565	-0.456548	6346.936035	1.089885	2093.416748	1784.469604	0.516715	00:24
47	1984.203735	2245.293701	2228.103271	16.879969	0.047386	0.808704	-0.498042	6430.269531	1.061489	2030.824341	1715.867310	0.510975	00:24
48	1979.119629	2352.370117	2333.208984	18.823326	0.047247	0.831529	-0.445145	6513.161133	1.102933	2110.088379	1818.141846	0.519061	00:23
49	1951.241577	2186.123779	2168.409912	17.496885	0.047719	0.818824	-0.473539	6246.245605	1.074579	2055.149414	1743.373291	0.510603	00:24
50	1928.953369	2226.338623	2209.330811	16.821451	0.041958	0.804572	-0.508660	6426.308594	1.056968	2022.071777	1707.827515	0.509897	00:23
51	1918.752075	2166.478516	2149.046387	17.217678	0.049111	0.804747	-0.509535	6236.624023	1.061681	2028.734375	1724.478760	0.514899	00:23
52	1903.347168	2149.011719	2130.229492	18.649336	0.054450	0.823058	-0.465431	6080.879883	1.092651	2093.164307	1797.216919	0.515208	00:24
53	1888.991211	2195.260742	2177.610352	17.639759	0.044792	0.817150	-0.477495	6222.992676	1.076117	2056.481201	1749.180542	0.507933	00:24
54	1881.612183	2141.744629	2124.793945	16.878689	0.047793	0.809202	-0.495584	6120.611816	1.057818	2021.609009	1703.235962	0.503281	00:23
55	1854.816528	2118.498291	2101.327881	17.104313	0.051836	0.810419	-0.493421	6000.839355	1.062874	2028.957764	1718.250610	0.505966	00:24
56	1839.432861	2258.764648	2241.761963	16.793386	0.044510	0.800588	-0.517331	6347.492676	1.051791	2010.334839	1695.131836	0.504905	00:24
57	1853.467041	2232.258545	2212.551025	19.500610	0.045772	0.834733	-0.436135	6296.918945	1.113015	2137.290771	1842.341064	0.510965	00:24
58	1837.273560	2078.458740	2061.684082	16.776508	0.045135	0.804238	-0.508361	6023.635742	1.054307	2008.124268	1699.466187	0.504561	00:24
59	1816.862427	2040.441772	2024.711670	15.597315	0.045059	0.785856	-0.552272	6029.678223	1.024677	1958.124878	1633.727783	0.496716	00:24
60	1819.208252	2139.017090	2121.553223	17.336550	0.055248	0.818801	-0.471689	6070.657227	1.071967	2050.424561	1734.173218	0.500660	00:23
61	1799.726440	2094.442139	2078.879150	15.395780	0.047046	0.782574	-0.561578	6207.199219	1.019686	1940.321533	1627.035156	0.499807	00:24
62	1783.477295	2189.658203	2173.854248	15.552533	0.050241	0.781507	-0.563103	6475.451172	1.019956	1941.290161	1626.447998	0.495016	00:24
63	1768.642334	2078.535889	2062.391846	15.885236	0.046041	0.790759	-0.541188	5978.875977	1.034647	1967.154175	1658.364868	0.500458	00:23
64	1764.174438	2000.245361	1984.001587	16.078659	0.050129	0.793963	-0.532840	5841.421875	1.038195	1977.391968	1664.403687	0.498570	00:24
65	1755.285645	2074.506592	2057.000977	17.236784	0.045309	0.807654	-0.500120	5898.439453	1.064845	2027.080688	1725.923340	0.503585	00:24
66	1751.253052	2192.845703	2176.969482	15.646471	0.047620	0.794049	-0.531214	6264.810547	1.032213	1964.389160	1644.656860	0.494554	00:24
67	1742.702148	2003.918945	1988.279419	15.603477	0.048556	0.792067	-0.536823	5857.966309	1.029384	1960.994385	1640.948608	0.496105	00:23
68	1734.003662	1960.304810	1944.389404	15.780478	0.042396	0.783507	-0.559525	5773.011719	1.025699	1953.704346	1641.869385	0.499045	00:23
69	1732.676270	2056.047363	2037.766235	17.803648	0.048374	0.805790	-0.505154	5835.386230	1.070438	2045.084351	1744.775269	0.504155	00:23
70	1719.732910	1968.918945	1953.201294	15.736311	0.049873	0.781208	-0.565390	5756.400391	1.023440	1944.613037	1639.710449	0.500073	00:24
71	1722.252686	2068.263672	2050.161621	17.727818	0.049945	0.812661	-0.488522	5823.361816	1.074614	2049.888184	1750.921021	0.505066	00:24
72	1715.597046	1943.568359	1928.785889	14.659951	0.044364	0.782311	-0.560319	5696.813965	1.011397	1924.081909	1601.036621	0.492042	00:23
73	1706.087524	2009.766113	1993.419189	16.229599	0.052192	0.795711	-0.528156	5771.199219	1.040488	1984.187622	1668.851074	0.497337	00:24
74	1690.334961	2119.673584	2104.496582	14.836791	0.047608	0.783031	-0.558614	6084.231445	1.013145	1926.117798	1605.451904	0.491907	00:24
75	1683.371094	1925.619385	1910.288330	15.424359	0.053724	0.778009	-0.573182	5706.276855	1.016423	1933.619019	1624.667114	0.498465	00:23
76	1671.691162	1938.888306	1923.586182	15.093276	0.047824	0.783145	-0.559653	5705.539551	1.016528	1929.590210	1617.487305	0.496029	00:23
77	1675.195557	1969.706665	1952.145996	17.220224	0.050449	0.801974	-0.512465	5722.925293	1.058287	2022.044189	1710.370239	0.497038	00:23
78	1667.553467	1951.343872	1935.015015	16.264315	0.045767	0.795914	-0.528531	5608.801270	1.041058	1982.651611	1671.711548	0.500002	00:23
79	1666.695801	1924.638428	1909.485229	14.979011	0.048244	0.778510	-0.570834	5615.051758	1.011054	1921.656250	1606.899292	0.493661	00:23
80	1667.953369	1927.399292	1912.782471	14.434852	0.048317	0.773981	-0.579921	5788.229492	1.000179	1900.219604	1578.418213	0.485158	00:23
81	1656.584839	1962.813721	1946.891479	15.511126	0.044743	0.791771	-0.538538	5702.780273	1.029768	1956.088745	1644.844849	0.498330	00:23
82	1637.365845	1897.853760	1882.795776	15.077114	0.045206	0.779908	-0.568294	5562.270996	1.011593	1918.007202	1609.076904	0.497358	00:23
83	1623.953735	1895.878540	1880.903564	14.739231	0.040865	0.766553	-0.603555	5562.432129	0.999141	1893.257202	1593.663086	0.499841	00:24
84	1631.195312	2004.035767	1989.149414	14.538733	0.047065	0.778626	-0.570778	5742.729004	1.006706	1907.639526	1596.172241	0.495295	00:23
85	1628.239136	1901.397583	1887.269165	13.971234	0.036782	0.762744	-0.610812	5755.364746	0.984519	1865.953613	1554.630371	0.490754	00:24
86	1635.655884	1890.822021	1876.614990	13.997304	0.047380	0.768836	-0.594105	5630.626465	0.990158	1881.327148	1560.892212	0.487769	00:23
87	1619.503906	1882.146240	1867.116211	14.936031	0.045019	0.769721	-0.595086	5506.448242	1.005221	1908.795898	1605.485474	0.498761	00:23
88	1619.595337	1880.541260	1865.143188	15.202730	0.049003	0.787871	-0.548113	5462.920898	1.022931	1941.532959	1630.943115	0.496960	00:23
89	1613.255249	1984.275024	1966.732910	17.282715	0.042405	0.804184	-0.508741	5638.116211	1.062198	2025.288452	1723.226807	0.501592	00:24
90	1606.169189	1862.778687	1847.071533	15.529962	0.044467	0.783225	-0.560873	5462.847656	1.022386	1939.350952	1636.303101	0.500584	00:23
91	1609.607544	1910.367065	1894.708618	15.496771	0.044426	0.790351	-0.541698	5494.456543	1.027497	1953.776001	1639.620239	0.496587	00:24
92	1603.590942	1960.099365	1946.069580	13.971668	0.046126	0.754995	-0.630675	5919.393555	0.977296	1855.564209	1545.912354	0.489731	00:24
93	1605.382080	1876.894043	1862.129883	14.676646	0.040245	0.775969	-0.576931	5583.471680	1.006059	1906.842041	1595.229858	0.492825	00:24
94	1583.398438	1937.457764	1921.093994	16.335154	0.041187	0.791778	-0.540101	5531.172852	1.039418	1977.000122	1674.374512	0.503145	00:23
95	1581.612549	1835.172241	1821.100098	13.890847	0.050193	0.765758	-0.603429	5450.705078	0.986337	1868.951904	1558.364624	0.491370	00:24
96	1579.690430	1837.527100	1822.542603	14.684216	0.048580	0.774344	-0.583328	5418.464355	1.004971	1905.832397	1600.656006	0.499638	00:24
97	1583.773926	1821.455322	1807.557617	13.652730	0.046067	0.761249	-0.615050	5497.828125	0.981084	1855.827515	1549.596802	0.491711	00:24
98	1583.227417	1819.533691	1804.496216	14.928608	0.041859	0.781699	-0.563601	5330.727051	1.012702	1914.940430	1609.623901	0.496521	00:23
99	1567.252808	1902.686035	1886.798462	15.758029	0.050320	0.780095	-0.570298	5583.282227	1.022527	1943.718628	1644.271240	0.505894	00:24
100	1572.749634	1863.321655	1849.039185	13.978075	0.041848	0.776276	-0.576076	5542.300293	0.998581	1885.867798	1576.105713	0.490765	00:24
101	1561.390869	1778.629150	1765.033936	13.473954	0.046851	0.760431	-0.617726	5365.498535	0.977076	1847.461060	1542.018799	0.493202	00:24
102	1554.405762	1793.989136	1780.624512	13.337927	0.039644	0.762962	-0.609675	5439.504883	0.976549	1845.786499	1533.581421	0.489440	00:24
103	1550.139282	1803.763306	1789.418335	14.204505	0.047718	0.777707	-0.573318	5340.282715	1.001259	1893.551270	1584.347534	0.495005	00:24
104	1545.177002	1808.277588	1794.598145	13.563501	0.045840	0.774189	-0.581362	5352.294434	0.989471	1872.578247	1555.489868	0.491507	00:23
105	1546.557739	1778.235474	1764.073730	14.111894	0.041205	0.778304	-0.571823	5251.641113	0.999497	1890.440552	1578.523682	0.494077	00:23
106	1537.222168	1781.041260	1767.304443	13.775725	0.041837	0.771026	-0.589410	5304.643066	0.988058	1865.651245	1554.898193	0.490515	00:24
107	1538.707764	1785.395874	1771.354004	13.960267	0.039857	0.780740	-0.565743	5323.761230	1.000003	1889.690552	1577.392700	0.494897	00:24
108	1533.052368	1833.327026	1819.516113	13.616963	0.046818	0.768385	-0.596994	5411.942383	0.986244	1861.312378	1555.800903	0.493678	00:23
109	1528.282227	1798.111816	1784.515137	13.528557	0.041809	0.762646	-0.611968	5357.743652	0.978249	1844.739990	1541.980103	0.493448	00:24
110	1526.862427	1799.446045	1786.277954	12.966899	0.043644	0.754512	-0.632367	5376.239258	0.964648	1818.849487	1516.334839	0.490012	00:24
111	1523.077637	1756.224487	1742.447998	13.529755	0.046487	0.771139	-0.589131	5274.266602	0.986443	1864.797241	1551.135742	0.490616	00:23
112	1529.325684	1797.935425	1784.176392	13.653956	0.040448	0.760077	-0.619595	5376.958984	0.978359	1844.459961	1547.438232	0.496279	00:24
113	1518.242676	1783.705078	1768.756226	14.822066	0.041936	0.775163	-0.582138	5282.527344	1.008120	1905.776001	1609.571045	0.501678	00:24
114	1520.407837	1912.749268	1897.477661	14.960381	0.051009	0.773271	-0.587717	5464.833496	1.008230	1902.993286	1614.846191	0.504838	00:24
115	1505.013062	1741.917847	1728.625122	13.149019	0.045469	0.759637	-0.619913	5271.270508	0.972376	1835.787842	1531.567139	0.493255	00:24
116	1509.686890	1916.331909	1900.899414	15.149902	0.051836	0.780137	-0.569676	5540.610352	1.016474	1923.016479	1627.815796	0.503041	00:24
117	1508.735962	1787.357178	1773.621582	13.524192	0.053513	0.768796	-0.595543	5362.060059	0.984265	1858.056152	1550.191162	0.492759	00:24
118	1501.052124	1791.610962	1778.476440	13.032609	0.045079	0.754121	-0.633713	5358.076660	0.965411	1822.041870	1519.513306	0.490795	00:24
119	1497.884155	1820.942505	1807.536865	13.198060	0.047576	0.752508	-0.639384	5370.798340	0.967550	1824.296631	1530.348755	0.496000	00:23
120	1499.869019	1731.496460	1718.068115	13.332547	0.045602	0.762194	-0.613499	5209.441895	0.977164	1840.784058	1541.098877	0.494337	00:24
121	1494.199707	1785.917114	1771.454224	14.315552	0.044095	0.782811	-0.561474	5258.367676	1.006539	1896.658691	1594.894897	0.498379	00:24
122	1484.143921	1752.403809	1739.713379	12.572518	0.042145	0.759958	-0.616125	5328.588867	0.963721	1820.042847	1502.684448	0.483027	00:24
123	1486.989014	1785.623657	1771.019897	14.418050	0.043136	0.774449	-0.583683	5241.328613	1.002226	1890.029541	1594.741089	0.501373	00:24
124	1491.107422	1840.187866	1827.505127	12.358881	0.037512	0.742803	-0.660973	5813.546387	0.946614	1792.139893	1481.054810	0.480716	00:24
125	1483.239990	1771.854858	1757.864868	13.799751	0.050438	0.777081	-0.574626	5345.870605	0.995722	1878.988159	1570.665894	0.493920	00:23
126	1480.210815	1763.597046	1750.544800	12.836942	0.038926	0.755063	-0.630958	5339.038574	0.963782	1815.727051	1513.101196	0.490371	00:24
127	1486.744751	1712.286255	1699.191284	13.088781	0.039298	0.765617	-0.604729	5259.208984	0.975931	1836.793945	1533.706909	0.493931	00:24
128	1471.546997	1744.941772	1731.547485	13.081504	0.039608	0.770892	-0.588619	5255.071289	0.980480	1847.018799	1533.461182	0.486240	00:24
129	1477.258301	1885.250488	1872.204834	12.834613	0.038478	0.752977	-0.632669	6258.566895	0.959195	1819.870117	1495.808105	0.476098	00:24
130	1476.063721	1722.854736	1709.984497	12.728988	0.046360	0.758009	-0.624227	5213.971680	0.966029	1821.071289	1518.164551	0.493540	00:23
131	1468.545532	1766.099487	1753.359375	12.661175	0.043363	0.762251	-0.613217	5275.878418	0.968005	1820.734619	1517.834229	0.493148	00:24
132	1460.417725	1691.528564	1678.043335	13.388208	0.044237	0.767331	-0.600559	5067.453613	0.981512	1850.208130	1547.028442	0.495345	00:24
133	1460.132812	1764.835205	1750.694824	13.836353	0.044856	0.778354	-0.572423	5155.833984	0.998214	1878.788574	1577.688232	0.496308	00:24
134	1449.852295	1717.159668	1703.481445	13.584293	0.045624	0.771395	-0.591638	5100.827148	0.988719	1863.547607	1564.609619	0.500363	00:24
135	1456.299805	1796.339844	1783.667358	12.612735	0.046080	0.750525	-0.642769	5302.807129	0.956472	1799.921875	1501.478394	0.488982	00:24
136	1452.066650	1712.855835	1700.749756	12.010898	0.035147	0.742716	-0.663375	5219.075195	0.942903	1773.968506	1477.056396	0.487644	00:24
137	1442.379883	1861.538940	1848.774536	12.546704	0.044301	0.757024	-0.625706	5465.744629	0.960394	1808.829956	1502.808716	0.489563	00:23
138	1451.181396	1740.815430	1727.772705	12.861029	0.046706	0.763278	-0.609869	5193.144531	0.970980	1828.928711	1522.987793	0.491339	00:24
139	1454.517456	1739.416748	1726.141968	13.127309	0.044694	0.763143	-0.611723	5189.341797	0.975750	1836.774536	1538.444946	0.495323	00:23
140	1453.098999	1809.774414	1796.948120	12.717151	0.037457	0.770682	-0.590425	5266.688965	0.975846	1836.441528	1524.875000	0.490269	00:24
141	1453.698975	1766.385376	1754.002197	12.168653	0.039576	0.760169	-0.614894	5354.198730	0.958532	1808.237793	1487.761963	0.480523	00:24
142	1436.513794	1740.355835	1727.632935	12.607018	0.044312	0.762749	-0.611572	5135.670410	0.967614	1823.881104	1515.742188	0.492147	00:24
143	1426.954102	1683.543213	1670.891602	12.386132	0.047308	0.758007	-0.623049	5045.751953	0.961070	1808.852173	1503.491089	0.489031	00:23
144	1435.731812	1737.902100	1725.594482	12.203060	0.038761	0.759287	-0.617784	5304.227539	0.958251	1806.407349	1489.541870	0.482690	00:23
145	1424.604614	1679.959961	1666.985229	13.009170	0.044784	0.765574	-0.604158	5029.961426	0.975828	1840.403076	1532.814331	0.491593	00:23
146	1430.846802	1745.726562	1733.709106	11.779364	0.041196	0.749969	-0.643417	5346.671875	0.946536	1781.426025	1475.560059	0.485592	00:24
147	1426.728638	1670.089111	1657.807495	12.069998	0.039616	0.753894	-0.633200	5110.452637	0.953304	1794.025757	1487.391113	0.486382	00:24
148	1421.082153	1770.327026	1758.205566	11.881845	0.031985	0.737488	-0.674429	5852.652344	0.935614	1771.917725	1459.603271	0.477787	00:24
149	1424.906250	1770.970581	1756.427368	14.251448	0.045018	0.778558	-0.571325	5154.751465	1.003317	1893.021362	1589.802979	0.494235	00:24
150	1428.417847	1722.283813	1709.760742	12.425946	0.044141	0.769665	-0.592743	5079.943359	0.971751	1825.433105	1515.940063	0.489371	00:24
151	1421.149902	1690.783936	1677.105225	13.628743	0.038097	0.767513	-0.600250	5135.013184	0.985277	1856.369751	1556.523438	0.495316	00:23
152	1433.208496	1708.456787	1696.220093	12.007226	0.042655	0.762878	-0.608496	5153.088867	0.959358	1806.983887	1488.737061	0.482379	00:24
153	1426.762085	1661.465698	1649.183472	12.254661	0.038874	0.767195	-0.597527	5165.588379	0.966164	1817.744263	1500.536743	0.483345	00:24
154	1413.873413	1764.028931	1751.354736	12.607476	0.041934	0.758214	-0.623549	5231.798828	0.962364	1810.543091	1508.350464	0.491851	00:24
155	1420.080811	1658.453003	1646.169556	12.203891	0.041853	0.761522	-0.614779	5024.637207	0.961713	1806.474243	1502.410034	0.492658	00:24
156	1413.686279	1679.697388	1667.682251	11.885459	0.037733	0.745325	-0.656726	5111.092773	0.943062	1773.028198	1475.005493	0.489388	00:24
157	1400.979492	1654.799316	1642.494995	12.223604	0.037701	0.755133	-0.631057	4976.752930	0.956361	1798.080322	1495.637329	0.489962	00:24
158	1401.699585	1685.698975	1673.149658	12.474750	0.043362	0.763767	-0.608483	5032.924805	0.966380	1817.423096	1510.497803	0.490456	00:24
159	1388.167847	1656.265991	1643.767822	12.446927	0.038312	0.761289	-0.615861	4995.916504	0.965072	1812.325317	1511.438354	0.493681	00:24
160	1384.202393	1649.645386	1637.485352	11.945498	0.044549	0.755672	-0.629659	4986.094238	0.953666	1792.926758	1489.143188	0.490317	00:23
161	1380.549316	1764.954346	1751.468750	13.289563	0.038319	0.755761	-0.630709	5191.483887	0.971387	1829.248657	1535.142456	0.494361	00:23
162	1375.546021	1743.503296	1732.010742	11.156135	0.041137	0.728400	-0.699738	5502.562500	0.918260	1733.994873	1433.800537	0.478874	00:23
163	1374.159058	1626.274414	1614.815918	11.381120	0.040864	0.735532	-0.682818	4987.481934	0.928477	1745.143433	1453.199219	0.487344	00:24
164	1373.824951	1683.784668	1672.551147	11.080626	0.036158	0.740005	-0.669310	5131.396484	0.927738	1745.729492	1441.605347	0.482697	00:23
165	1371.072754	1615.743530	1604.103882	11.577813	0.041652	0.736403	-0.681210	4918.139648	0.931138	1747.698853	1460.021118	0.490530	00:24
166	1355.484497	1641.220337	1630.196533	10.931018	0.036010	0.728178	-0.700613	5175.681641	0.913081	1720.974487	1422.012451	0.479188	00:24
167	1359.108276	1704.549927	1693.549438	10.845896	0.037629	0.718955	-0.726589	5429.520508	0.905963	1707.895020	1420.713013	0.481644	00:23
168	1356.763306	1649.856689	1638.938843	10.599342	0.041183	0.716552	-0.732771	5240.663086	0.901213	1699.459473	1412.924438	0.479067	00:24
169	1349.421021	1664.265259	1652.907837	11.272820	0.044008	0.724586	-0.713310	5020.294434	0.916043	1721.019531	1441.487793	0.488975	00:24
170	1350.994751	1623.858765	1612.932129	10.863392	0.041908	0.722831	-0.716796	5034.023926	0.908996	1710.285278	1424.442871	0.484604	00:24
171	1351.224976	1680.315918	1669.719727	10.547542	0.039740	0.710513	-0.750267	5435.653809	0.893453	1683.916748	1405.795776	0.481605	00:24
172	1346.770630	1604.967896	1593.335571	11.680385	0.039912	0.727918	-0.705782	4868.971191	0.924622	1739.492676	1460.044800	0.494222	00:24
173	1342.060303	1586.324097	1575.160645	11.029417	0.039247	0.720196	-0.726324	4858.954102	0.910087	1709.755005	1435.786011	0.491581	00:24
174	1338.160156	1588.046753	1577.109497	10.771853	0.035214	0.718428	-0.730816	4868.910156	0.906129	1701.200806	1427.917358	0.490517	00:24
175	1332.239746	1641.543823	1629.458740	11.952484	0.044998	0.719480	-0.730327	4942.689453	0.922858	1739.966675	1472.141724	0.498147	00:24
176	1325.793579	1655.992920	1643.510742	12.296820	0.043069	0.721262	-0.726518	4958.390625	0.929458	1751.547485	1487.265381	0.502298	00:24
177	1320.107178	1602.318115	1592.141968	9.982615	0.040755	0.695351	-0.794835	4962.369629	0.875475	1646.546143	1393.858398	0.486541	00:24
178	1316.732788	1599.840820	1589.633057	10.056231	0.039106	0.693247	-0.800047	5112.582520	0.871944	1642.347046	1387.680298	0.483511	00:24
179	1316.269165	1575.963501	1565.661133	10.261293	0.040414	0.692616	-0.804281	4894.453125	0.875071	1645.201904	1401.043335	0.491689	00:24
180	1312.937378	1565.147705	1554.971313	10.095403	0.039648	0.688514	-0.816473	4870.932129	0.870049	1635.080322	1397.676392	0.492972	00:23
181	1310.258423	1560.136353	1550.039917	10.130189	0.040148	0.689681	-0.813038	4840.253906	0.870697	1635.572021	1396.965210	0.492491	00:24
182	1303.114990	1559.687256	1549.733643	9.865709	0.039047	0.681105	-0.837071	4862.216797	0.858795	1614.653198	1384.084106	0.489868	00:24
183	1298.939209	1550.950439	1541.101929	9.818651	0.037294	0.678011	-0.846824	4839.537109	0.855971	1607.928955	1384.890259	0.491908	00:24
184	1296.500000	1558.706543	1548.998169	9.687429	0.041066	0.672073	-0.864892	4854.323242	0.849401	1596.333740	1383.215210	0.493486	00:24
185	1293.234863	1547.689209	1537.501587	10.014702	0.038959	0.676473	-0.853218	4749.744141	0.860303	1615.902466	1401.379639	0.498134	00:24
186	1292.811646	1546.130127	1535.702148	10.226321	0.042530	0.678689	-0.847562	4762.799316	0.865040	1625.793945	1409.895508	0.501073	00:24
187	1290.489868	1535.805420	1526.125854	9.580198	0.038133	0.669057	-0.874888	4788.103027	0.846551	1589.678711	1384.710449	0.496846	00:23
188	1286.943726	1534.413452	1524.609619	9.683949	0.039481	0.668009	-0.878814	4777.341797	0.847518	1590.329102	1390.739990	0.499342	00:23
189	1284.725708	1535.779663	1525.948364	9.720008	0.038524	0.666964	-0.882472	4757.062012	0.847355	1589.900513	1393.358276	0.501523	00:24
190	1281.207520	1533.668091	1524.209351	9.349226	0.037252	0.661511	-0.897568	4794.526855	0.836779	1571.197021	1378.563721	0.496789	00:23
191	1281.659302	1533.689453	1523.680420	9.959739	0.038388	0.669413	-0.875679	4734.559082	0.852867	1600.474243	1401.654541	0.502732	00:24
192	1277.832642	1525.408081	1515.864014	9.418250	0.037457	0.659884	-0.903180	4768.744629	0.836689	1569.301270	1383.119507	0.499142	00:24
193	1280.964233	1526.267944	1516.641357	9.531398	0.038529	0.657949	-0.910114	4741.918457	0.836838	1569.322144	1389.669189	0.502961	00:24
194	1272.813477	1520.035156	1510.463379	9.607328	0.037974	0.659625	-0.904838	4727.374023	0.838028	1571.622803	1388.355347	0.502201	00:23
195	1269.634644	1519.734131	1510.097290	9.620123	0.039231	0.658118	-0.909451	4724.170410	0.836997	1570.078369	1389.426147	0.502542	00:23
196	1277.440552	1518.647949	1509.047485	9.579507	0.038892	0.658164	-0.909332	4728.376953	0.836522	1568.702026	1388.311279	0.502341	00:24
197	1281.807983	1518.112183	1508.432251	9.564679	0.039367	0.658109	-0.909897	4729.467285	0.837224	1569.640137	1390.796387	0.503976	00:24
198	1277.494141	1518.218018	1508.575928	9.431640	0.039109	0.655921	-0.916428	4730.639648	0.834341	1564.817627	1388.831421	0.503389	00:24
199	1279.022827	1517.873779	1508.196777	9.541930	0.039074	0.656656	-0.914409	4725.548828	0.835692	1566.833374	1390.649292	0.504218	00:24

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	43863.750000	38415.714844	38403.582031	4.531198	-0.005897	0.299329	-2.509599	52373.148438	0.443983	215.868622	578.306458	0.593813	00:24
1	29051.519531	25556.123047	25545.445312	6.811570	-0.004425	0.374405	-2.064293	41032.382812	0.560199	277.017273	490.831116	0.606257	00:23
2	20931.611328	19844.193359	19834.179688	7.637002	-0.004937	0.397987	-1.930601	35639.593750	0.592585	290.151154	466.145905	0.562413	00:24
3	14663.121094	12426.390625	12416.396484	8.592687	-0.002675	0.421610	-1.805782	27062.740234	0.626879	308.991547	445.806946	0.527288	00:24
4	10193.139648	9417.468750	9406.019531	10.110537	-0.011966	0.455588	-1.648575	22221.794922	0.682492	335.222321	429.319397	0.516746	00:23
5	7524.279785	6364.255859	6352.415039	10.702779	0.011146	0.471224	-1.580377	17584.408203	0.701387	342.728180	421.015381	0.519270	00:23
6	6115.612305	5585.450684	5572.469727	11.632233	-0.000796	0.494548	-1.479531	15968.841797	0.733156	358.945038	410.607941	0.503936	00:23
7	5255.900879	5021.166016	5008.228516	11.662443	-0.007470	0.507035	-1.426110	14532.230469	0.745946	364.898315	403.718201	0.487873	00:23
8	4705.863281	4692.226562	4679.104004	11.865071	0.000847	0.522057	-1.364778	13638.712891	0.759972	370.586456	396.001831	0.477372	00:24
9	4339.275391	4509.605469	4495.935059	12.448554	-0.006662	0.537097	-1.307204	13147.251953	0.781810	380.401398	393.854126	0.474533	00:23
10	4063.217041	4339.301758	4325.839355	12.518625	-0.000959	0.539867	-1.295752	12419.667969	0.782707	379.506470	391.445526	0.469356	00:24
11	3844.226074	3915.171143	3901.546631	12.703267	-0.002186	0.557496	-1.232466	11357.519531	0.798980	386.335205	386.182617	0.474052	00:24
12	3663.054199	4350.450684	4336.080566	13.246813	-0.008105	0.564001	-1.208974	11926.161133	0.812054	393.216003	387.989990	0.474185	00:24
13	3521.080322	3717.507324	3703.758057	12.813422	-0.001015	0.569932	-1.187693	10647.582031	0.811043	391.481812	382.930420	0.473800	00:23
14	3417.034912	3798.777100	3783.877441	14.170590	-0.016727	0.591292	-1.114504	10922.300781	0.846533	408.519867	386.855652	0.475482	00:24
15	3295.059814	3525.010742	3510.413818	13.814376	-0.007868	0.580232	-1.151974	10026.403320	0.834055	402.132599	387.572205	0.472764	00:24
16	3211.870850	3535.393066	3520.083984	14.482000	-0.007838	0.606115	-1.062313	9636.927734	0.860866	415.887024	384.029480	0.462664	00:24
17	3143.329590	3317.885498	3303.194824	14.088883	-0.001076	0.601462	-1.077590	9092.801758	0.851771	410.015045	381.872803	0.463364	00:24
18	3076.614990	3438.985596	3424.058838	14.395399	0.000178	0.608181	-1.056857	9169.375000	0.863004	413.457825	384.296448	0.470794	00:24
19	3009.324219	3417.989014	3403.146240	14.122858	-0.007747	0.606345	-1.062125	9251.507812	0.858551	413.128448	382.520508	0.467891	00:23
20	2935.841309	3159.706299	3144.865479	14.208220	-0.003280	0.608992	-1.053701	8666.197266	0.862335	413.299194	383.190063	0.468349	00:23
21	2889.335449	3118.774902	3104.639648	13.635227	0.006028	0.614202	-1.034251	8542.797852	0.857498	410.562469	376.163757	0.459089	00:23
22	2904.316650	3182.259766	3166.236572	15.280206	-0.009078	0.650401	-0.921584	8301.858398	0.909851	435.520782	386.079926	0.463378	00:23
23	2842.148438	3260.038330	3245.093750	14.698691	-0.003224	0.629414	-0.985648	8522.114258	0.881078	422.249756	380.361420	0.460035	00:24
24	2805.152588	3302.236084	3285.666748	16.182764	-0.007153	0.645778	-0.935025	8513.980469	0.916546	439.027618	391.966736	0.460427	00:24
25	2785.724121	3376.423096	3359.880615	15.800976	-0.001768	0.657435	-0.901159	8544.414062	0.920507	438.538971	389.170227	0.466704	00:24
26	2762.451416	2964.311523	2949.358643	14.561893	-0.000443	0.647257	-0.929403	8000.426758	0.894137	427.106110	377.525421	0.458053	00:23
27	2724.360596	2976.501221	2960.346680	15.885023	-0.002973	0.651166	-0.919243	7846.898926	0.915923	437.352905	388.857391	0.463826	00:24
28	2703.101807	2907.522461	2892.239502	14.919702	-0.004438	0.654061	-0.908257	7749.744629	0.905100	431.247742	380.027710	0.455881	00:23
29	2670.739502	2896.980957	2881.499268	14.934103	-0.003205	0.652672	-0.913074	7585.269531	0.907845	433.303772	382.666962	0.456568	00:23
30	2657.177246	2908.962158	2893.283447	15.238147	0.002911	0.667215	-0.869191	7627.063477	0.921041	438.859680	382.833466	0.459951	00:24
31	2632.270264	2971.827881	2956.658691	14.708002	0.000174	0.651452	-0.916749	7723.069336	0.901899	430.017609	379.759186	0.459874	00:23
32	2633.249023	2896.732422	2881.563477	14.930392	0.000247	0.654663	-0.906587	7833.086426	0.906260	431.945465	380.294464	0.458233	00:23
33	2599.547119	2908.443604	2893.063965	14.918993	0.005476	0.661886	-0.884253	7560.501953	0.911457	433.857971	379.544250	0.454524	00:24
34	2592.372803	2866.823975	2851.997314	13.981042	0.007245	0.658070	-0.894802	7673.434082	0.896489	426.589722	372.266479	0.451590	00:24
35	2572.281494	2804.459961	2789.447754	14.607452	0.005032	0.661891	-0.885020	7414.482910	0.908774	433.066833	378.309479	0.458576	00:24
36	2559.514160	2804.730713	2789.944824	14.492640	0.009472	0.655917	-0.902291	7479.042969	0.900253	428.058014	376.270325	0.454680	00:24
37	2549.586182	2803.934570	2788.795410	14.567461	0.001700	0.673275	-0.849742	7306.768555	0.917193	435.740265	377.050690	0.453040	00:24
38	2542.187500	2838.276611	2822.072754	15.779968	0.002494	0.678474	-0.836513	7458.104004	0.938841	446.885040	388.480835	0.460749	00:23
39	2528.304443	2727.795166	2712.283447	15.043669	0.010385	0.680495	-0.828586	7119.888672	0.929208	441.239685	380.757141	0.454719	00:23
40	2509.762939	2745.618652	2730.799072	14.620611	0.006320	0.670531	-0.857315	7317.892578	0.913483	434.135284	375.863922	0.451071	00:24
41	2498.480957	2984.975830	2968.752930	15.645418	0.002487	0.689347	-0.803794	7466.937500	0.944765	448.014771	386.595825	0.455587	00:23
42	2526.743652	2784.272705	2767.412354	16.525219	0.008349	0.699563	-0.774746	7292.501465	0.962843	458.175140	393.207031	0.458788	00:24
43	2479.489502	2716.605225	2701.625977	14.440147	0.008998	0.675520	-0.843820	7103.064453	0.916906	435.091675	376.118988	0.454560	00:24
44	2450.944580	2820.361084	2805.691895	14.351919	-0.000299	0.681099	-0.825373	7175.885742	0.918579	435.982666	373.253693	0.447491	00:24
45	2399.970947	2855.157227	2840.262939	14.562766	-0.006594	0.668939	-0.862151	7546.999023	0.912756	435.205444	376.112946	0.453263	00:24
46	2318.381836	2948.972656	2933.513184	14.931308	-0.010820	0.669663	-0.858805	7707.319336	0.917600	438.631836	377.986450	0.446189	00:24
47	2233.905518	2531.709229	2517.131104	14.154142	-0.004406	0.668042	-0.863950	6903.244629	0.908768	432.607391	373.857697	0.447973	00:24
48	2210.590088	2912.146973	2897.474609	14.289447	0.000908	0.672408	-0.850297	7493.521973	0.910653	433.871857	372.607941	0.444153	00:24
49	2193.390869	2784.932617	2770.388428	14.446658	0.002993	0.675446	-0.841248	7174.827637	0.914766	435.479675	373.613892	0.444635	00:24
50	2167.483643	2572.924316	2558.306152	14.623973	-0.004008	0.672136	-0.852110	7034.622070	0.915774	435.759369	376.101990	0.448337	00:24
51	2148.242920	2437.702881	2422.453613	14.876254	0.001187	0.677917	-0.836370	6642.927734	0.924982	439.490601	379.451447	0.453160	00:23
52	2119.804688	2564.101807	2549.729980	14.138188	0.008252	0.682225	-0.822155	6762.032227	0.917301	434.697266	372.190369	0.446474	00:24
53	2106.597412	2382.819824	2368.285400	14.838985	0.008345	0.682144	-0.822582	6569.984863	0.921236	437.164886	374.518799	0.449668	00:23
54	2104.226807	2605.211182	2589.143311	15.944120	0.000012	0.704920	-0.759345	6805.437500	0.958070	453.829285	387.741425	0.458698	00:24
55	2091.646240	2432.240479	2417.528320	14.472643	0.014744	0.687805	-0.807240	6672.047363	0.926808	438.970825	375.887177	0.453741	00:24
56	2077.431396	2535.636475	2520.333496	15.055017	0.007748	0.706880	-0.751748	6741.846680	0.947529	448.772888	379.493866	0.449344	00:23
57	2097.115723	2535.762939	2520.549561	15.058545	0.010610	0.698099	-0.775957	6880.807129	0.943769	447.930084	380.545807	0.447572	00:24
58	2074.907715	2496.313721	2480.572754	15.564722	-0.000937	0.702782	-0.763651	6622.996582	0.953701	452.134033	385.031403	0.450153	00:24
59	2062.163086	2475.023193	2460.586914	14.161969	0.014412	0.689498	-0.801185	6833.764648	0.923114	436.862000	372.420868	0.448638	00:24
60	2049.748779	2374.473389	2359.209473	15.079268	0.011797	0.700543	-0.771383	6404.564941	0.945152	447.325958	381.644592	0.455207	00:24
61	2047.361084	2376.252441	2360.949707	15.071036	0.001268	0.705669	-0.756095	6325.954102	0.950458	449.741394	382.154999	0.452023	00:23
62	2030.971680	2297.718750	2283.258545	14.449381	0.006626	0.690856	-0.795822	6350.029785	0.926574	439.087372	372.843842	0.442442	00:23
63	2031.379517	2573.683350	2560.023438	13.541169	0.002363	0.680560	-0.826074	6887.925781	0.907360	429.354614	366.436218	0.443858	00:23
64	2022.141724	2335.422607	2320.044678	15.109181	0.009062	0.710958	-0.741824	6309.033691	0.955586	450.358704	383.545898	0.454912	00:24
65	2043.625610	2400.428955	2384.999512	15.008455	0.003341	0.713824	-0.731893	6451.890625	0.956545	453.298187	381.855011	0.446871	00:24
66	2024.132690	2468.580322	2452.721924	15.564773	0.003289	0.714655	-0.729227	6588.416992	0.962702	455.840393	384.967712	0.446715	00:24
67	1996.843994	2270.792725	2255.573242	14.917750	0.015514	0.715168	-0.729083	6179.713379	0.956190	451.436707	381.625122	0.453211	00:23
68	1996.723633	2653.511475	2638.159912	15.123527	0.009264	0.713429	-0.732334	6925.761719	0.956444	452.881958	381.582092	0.445393	00:24
69	1993.352417	2318.995850	2303.337158	15.534203	0.010102	0.723970	-0.703007	6181.556641	0.968995	457.881866	385.096649	0.445595	00:24
70	1997.632080	2365.313721	2351.059570	14.203370	0.013818	0.712918	-0.733379	6572.615723	0.942815	446.432953	373.483856	0.444729	00:24
71	1991.968140	2434.370850	2418.563477	15.534522	0.012001	0.728581	-0.691676	6287.305176	0.973917	460.384094	387.257904	0.450199	00:23
72	1984.693359	2430.349365	2415.005127	15.024178	0.005615	0.722411	-0.708394	6508.200684	0.964146	455.255981	383.346558	0.449675	00:24
73	1972.730835	2342.661621	2327.041504	15.494254	0.001184	0.722117	-0.709227	6325.833008	0.966910	457.598572	385.087433	0.448399	00:24
74	1981.359619	2544.892578	2528.841309	15.822600	0.009106	0.729326	-0.688972	6672.772949	0.978393	461.669189	389.258881	0.448416	00:23
75	1969.436523	2298.509766	2283.355957	15.099205	0.011240	0.722553	-0.707532	6332.187988	0.964057	455.776276	383.052948	0.446579	00:24
76	1951.999146	2364.954102	2349.126953	15.702997	0.008855	0.733929	-0.677247	6368.576660	0.980469	462.792969	389.344727	0.449776	00:24
77	1951.475342	2224.091797	2208.672607	15.312521	0.008498	0.733552	-0.676775	6025.102051	0.974773	460.427124	384.907806	0.446509	00:24
78	1957.876831	2426.103027	2411.799316	14.131270	0.007124	0.729588	-0.686136	6351.814941	0.955903	451.659302	374.227509	0.439011	00:23
79	1941.185791	2294.367676	2279.660645	14.518409	0.023894	0.723170	-0.703930	6132.000488	0.955547	452.413910	376.670837	0.441308	00:24
80	1943.570068	2236.377197	2221.678955	14.449833	0.007350	0.724631	-0.700530	6133.900391	0.957083	451.618469	377.185211	0.443405	00:24
81	1947.266846	2247.182373	2232.464355	14.676343	0.003813	0.726550	-0.694701	6146.789551	0.958115	452.276642	376.816711	0.441424	00:23
82	1957.837280	2534.509277	2520.347168	14.056922	0.016048	0.724160	-0.702193	6636.850098	0.950278	448.406555	373.571930	0.445898	00:24
83	1933.469727	2579.322266	2563.413086	15.722889	0.011245	0.737353	-0.666855	6630.823730	0.981821	463.296265	388.400604	0.447950	00:23
84	1942.276489	2363.916992	2348.039551	16.134193	0.012214	0.746446	-0.644253	6304.797852	0.994142	468.833160	393.955200	0.454514	00:24
85	1928.095215	2209.769043	2194.548340	15.230845	0.008082	0.734415	-0.675613	5984.231445	0.973779	459.524414	384.817932	0.450239	00:24
86	1921.092529	2302.175049	2287.004883	14.949891	0.009675	0.735890	-0.670195	6158.293457	0.971037	457.397339	381.883636	0.445640	00:24
87	1910.358398	2239.969727	2225.645508	14.069258	0.005961	0.724680	-0.700270	6194.545898	0.951774	448.981903	373.757874	0.443217	00:23
88	1922.070801	2302.062256	2286.641846	15.184399	0.019390	0.741756	-0.655016	6103.609863	0.977987	460.710144	384.518677	0.448399	00:24
89	1914.837036	2212.521240	2197.232910	15.218254	0.012381	0.749186	-0.635606	6004.154785	0.987106	464.512451	387.790405	0.451465	00:24
90	1908.945190	2239.150879	2222.606445	16.385725	0.010513	0.752060	-0.627548	6045.464355	1.004142	474.579285	397.498474	0.447366	00:23
91	1919.256104	2281.434326	2266.269043	15.264541	0.014341	0.755768	-0.616070	6124.239258	0.989729	466.477112	386.233459	0.442961	00:23
92	1900.793091	2262.195312	2245.846924	16.261921	0.009571	0.757675	-0.613236	5989.441406	1.008500	475.759094	398.756073	0.449456	00:23
93	1896.890991	2302.918213	2287.291748	15.300570	0.016387	0.750865	-0.629952	6144.000977	0.989973	466.276306	388.559601	0.445447	00:23
94	1889.807983	2357.636475	2342.659668	15.078780	0.010196	0.759209	-0.606814	6270.000488	0.991948	468.093231	386.461212	0.441518	00:24
95	1909.732300	2252.490234	2236.052490	16.184378	0.006094	0.758564	-0.610889	6073.866699	1.007757	474.724731	397.968323	0.451051	00:24
96	1888.111328	2201.737061	2186.147705	15.409701	0.005468	0.755066	-0.618677	5916.775391	0.993156	467.651886	389.073517	0.443881	00:24
97	1876.192383	2205.862305	2190.198975	15.518577	0.013440	0.755652	-0.618026	5960.430664	0.994487	467.427460	390.132782	0.448773	00:24
98	1891.700195	2235.016113	2218.613037	16.125742	0.010912	0.763022	-0.599383	5976.390137	1.011120	476.187958	398.826996	0.451252	00:23
99	1879.114868	2160.450928	2144.865234	15.580759	0.009719	0.757886	-0.611456	5865.650391	0.997542	469.790405	390.953796	0.446132	00:24
100	1878.698120	2264.475830	2250.002197	14.256703	0.019321	0.759118	-0.605667	6185.930664	0.982463	463.984253	379.972717	0.435502	00:23
101	1884.965210	2276.707764	2260.010742	16.751415	0.018522	0.774234	-0.570665	6064.729980	1.025500	482.417511	405.142883	0.452555	00:23
102	1868.320190	2195.005127	2179.216309	15.626679	0.018922	0.765999	-0.590748	5919.426270	1.005778	472.568909	394.234589	0.448182	00:24
103	1870.577271	2325.707764	2310.781250	14.914661	0.016117	0.764631	-0.593082	6181.680176	0.992466	467.353271	385.482513	0.442336	00:24
104	1861.645630	2164.401123	2149.490479	14.860444	0.010910	0.763384	-0.596666	5896.706543	0.992715	467.017334	385.989929	0.445966	00:24
105	1854.688232	2361.593750	2345.624756	15.899466	0.002891	0.776262	-0.563304	6310.805664	1.016886	478.773987	397.732208	0.443036	00:23
106	1858.104004	2166.645020	2151.011963	15.698688	0.003630	0.767692	-0.584930	5854.079102	1.005505	474.027863	392.670990	0.441261	00:23
107	1861.227051	2194.676758	2179.235107	15.386620	0.012783	0.762215	-0.599076	5973.272949	0.997356	469.537842	388.933960	0.442284	00:24
108	1853.402222	2214.352051	2197.413086	16.974899	0.008637	0.774503	-0.568865	5948.845703	1.028581	483.793304	406.355316	0.448923	00:24
109	1853.154053	2219.520508	2202.921387	16.433540	0.006969	0.773090	-0.571207	5958.150391	1.022447	482.378448	402.077209	0.442657	00:23
110	1844.206543	2221.476562	2206.081055	15.232481	0.017018	0.776417	-0.561965	5881.692871	1.007130	474.478088	390.759552	0.442425	00:24
111	1847.569580	2295.849365	2277.609619	18.011066	0.013199	0.793577	-0.520873	6022.988281	1.058052	498.998871	421.094086	0.450569	00:24
112	1844.456665	2218.931641	2203.596680	15.319036	0.012254	0.774684	-0.565667	6142.659180	1.008328	474.474670	391.739655	0.438111	00:23
113	1841.190552	2190.828613	2173.822998	16.907028	0.014819	0.792422	-0.522674	5820.849609	1.043826	491.324738	411.315399	0.446875	00:24
114	1835.547241	2351.720947	2336.396729	14.853375	0.018431	0.769720	-0.579932	6318.644043	1.000530	470.900970	389.046448	0.444679	00:24
115	1824.982056	2149.504150	2133.460205	16.047073	0.014818	0.784466	-0.541972	5829.170898	1.025810	483.199127	400.937317	0.443612	00:24
116	1829.949829	2174.595215	2159.585938	15.228032	0.007764	0.768903	-0.580901	6015.817383	1.000651	470.963165	388.486206	0.439754	00:24
117	1828.137207	2159.846191	2143.157959	16.303429	0.016402	0.774903	-0.568036	5800.884277	1.023074	480.765625	402.793121	0.450413	00:24
118	1821.919312	2172.395264	2157.207275	15.308952	0.022732	0.784529	-0.540608	5832.681641	1.012731	476.492767	391.950256	0.439437	00:23
119	1823.590820	2124.030029	2108.170166	16.110052	0.017425	0.787079	-0.534081	5774.553223	1.026570	483.151154	400.051453	0.439420	00:24
120	1816.651489	2134.261719	2118.056396	16.119879	0.017726	0.785489	-0.540190	5736.674805	1.027853	483.374786	402.598602	0.446004	00:24
121	1825.160400	2303.302979	2286.981445	16.150137	0.011127	0.790506	-0.526535	6017.223633	1.031576	485.434357	403.089417	0.442133	00:24
122	1812.840454	2192.228027	2177.015869	15.022361	0.017740	0.783207	-0.543806	5906.555176	1.012327	476.290436	391.977814	0.437572	00:24
123	1813.112915	2162.520996	2147.565430	15.085690	0.018595	0.787382	-0.533425	5929.552246	1.015461	477.697510	392.893707	0.439185	00:24
124	1811.771484	2091.887695	2075.936523	15.922107	0.012522	0.776765	-0.561986	5759.767578	1.015987	477.538086	397.007172	0.444426	00:24
125	1823.553345	2352.743408	2337.209717	15.474758	0.005156	0.791367	-0.523368	6327.042480	1.024082	481.934967	397.547821	0.436857	00:24
126	1805.034668	2180.511963	2164.601562	15.944650	0.011578	0.784811	-0.541834	5812.856934	1.026006	482.167419	401.474640	0.445950	00:24
127	1807.343872	2181.065918	2165.801270	15.445350	0.012048	0.786192	-0.537312	5858.233398	1.020407	479.858826	396.748749	0.442926	00:24
128	1801.989502	2138.415527	2122.885498	15.467233	0.016893	0.791176	-0.523486	5808.321777	1.023366	481.355225	396.787476	0.438674	00:24
129	1801.509277	2315.857666	2299.096436	16.754320	0.014647	0.804377	-0.493137	6029.151367	1.051027	493.211853	413.338501	0.448887	00:24
130	1797.859985	2107.056885	2091.309082	15.413078	0.011560	0.798054	-0.506407	5679.422852	1.030524	484.914032	399.735352	0.438550	00:24
131	1798.474731	2293.315430	2278.733154	14.416295	0.004536	0.777620	-0.555554	6227.265625	0.999133	472.173157	383.337585	0.428521	00:23
132	1803.351074	2255.859619	2239.011475	16.752981	0.010490	0.802543	-0.498135	5911.997070	1.048473	492.301392	412.187164	0.449776	00:23
133	1802.076660	2212.726318	2196.369629	16.381973	0.015475	0.795316	-0.514360	5925.979004	1.036741	486.914734	405.199432	0.443403	00:24
134	1778.235596	2241.857422	2226.428223	15.551047	0.016160	0.792991	-0.520058	5921.941406	1.024173	481.234497	397.460571	0.442906	00:24
135	1798.828857	2215.352783	2199.614746	15.520594	-0.000083	0.798767	-0.504301	5998.493164	1.031573	484.854431	399.885162	0.437686	00:24
136	1791.604004	2172.814941	2156.524902	16.056366	0.007524	0.799264	-0.503785	5884.136230	1.040192	489.181946	405.979462	0.440864	00:25
137	1795.246948	2234.702148	2219.749023	14.842214	0.014538	0.800434	-0.499434	6144.828613	1.022123	480.883423	393.029816	0.435652	00:23
138	1793.262695	2083.352051	2067.292236	15.950337	0.011878	0.798019	-0.507013	5692.207031	1.035376	486.541351	403.273041	0.440748	00:24
139	1783.274414	2149.961670	2133.359131	16.134953	0.010936	0.801305	-0.499190	5738.727539	1.042928	490.078369	407.561859	0.442594	00:24
140	1789.629883	2315.940430	2297.484863	18.563274	0.007493	0.817955	-0.460288	5939.103027	1.081770	507.795471	431.535492	0.449273	00:23
141	1791.627930	2207.731201	2191.982666	15.709380	0.013139	0.822607	-0.445180	5963.995117	1.051367	495.090454	407.621033	0.436277	00:23
142	1786.481079	2178.852295	2164.209961	14.598886	0.018185	0.781908	-0.545324	6043.077637	1.001952	473.065735	384.565674	0.433211	00:24
143	1772.505615	2088.655273	2072.413574	16.312189	0.009611	0.806352	-0.487083	5623.616211	1.048144	492.382812	410.063354	0.443864	00:23
144	1773.765015	2246.583984	2229.790283	16.737934	0.010199	0.801941	-0.498106	5920.999023	1.049527	492.969116	412.185272	0.444252	00:24
145	1781.379272	2127.945312	2112.522705	15.241892	0.014805	0.808196	-0.481105	5745.976074	1.035487	486.594574	400.525848	0.439721	00:23
146	1775.792358	2542.838867	2527.221924	15.221790	0.017093	0.815300	-0.461849	6359.957031	1.040990	490.206329	401.406860	0.432750	00:23
147	1774.002686	2196.975098	2178.918457	17.970129	0.008381	0.819415	-0.454260	5786.255859	1.077251	507.290497	426.548492	0.440215	00:24
148	1763.355225	2454.446777	2438.522461	15.798021	0.018277	0.814125	-0.465027	6367.140625	1.044590	492.299805	404.380280	0.431745	00:24
149	1763.413696	2331.704346	2316.855713	14.775893	0.017670	0.800866	-0.497773	6180.981934	1.020494	481.215607	391.676636	0.433188	00:23
150	1766.779297	2101.832275	2085.813721	15.988742	0.010703	0.805431	-0.485994	5637.121094	1.041236	490.133606	403.671326	0.431689	00:24
151	1756.943115	2089.755859	2074.065186	15.755790	0.013304	0.808495	-0.479667	5680.389160	1.041188	489.413513	403.752808	0.435701	00:23
152	1757.968384	2153.202393	2137.951660	15.111240	0.014387	0.802591	-0.493400	5782.798340	1.028324	484.404449	396.194733	0.431953	00:24
153	1761.133423	2126.274170	2108.299316	18.150089	0.015727	0.824693	-0.441610	5691.865234	1.081061	508.173676	428.076447	0.442108	00:23
154	1754.218750	2100.638672	2085.303955	15.198414	0.013400	0.813222	-0.467146	5764.148926	1.037852	488.758392	400.037048	0.431861	00:23
155	1752.608276	2185.967529	2169.387451	16.396458	0.010934	0.819536	-0.452934	5783.447266	1.059863	498.460938	413.928986	0.436640	00:23
156	1761.809570	2086.500244	2070.962891	15.445103	0.017514	0.817518	-0.457789	5658.406738	1.046739	492.070557	405.519073	0.438143	00:23
157	1744.872314	2104.696533	2087.964355	16.573210	0.009364	0.818238	-0.456819	5663.222168	1.060570	498.739502	415.193909	0.439059	00:24
158	1755.324707	2210.587646	2191.904297	18.642618	0.007834	0.829451	-0.430653	5912.790527	1.094097	515.484314	436.298737	0.444831	00:23
159	1750.736938	2276.606934	2261.609619	14.890740	0.013379	0.804139	-0.489871	6127.239258	1.025618	482.895538	394.194458	0.435201	00:23
160	1745.393188	2089.162354	2074.119873	15.043243	0.015280	0.801667	-0.495509	5638.349609	1.024409	481.983551	393.741150	0.431860	00:24
161	1736.090942	2147.017822	2129.102783	17.569410	0.008687	0.817546	-0.459487	5712.206543	1.073218	504.761566	424.593384	0.442993	00:23
162	1738.413940	2367.807373	2351.743164	16.021133	0.013200	0.809847	-0.476627	6164.467773	1.043733	490.112000	405.359436	0.437047	00:24
163	1730.273926	2095.576904	2079.865723	15.692489	0.015031	0.804401	-0.490447	5642.889648	1.037469	487.596954	402.597015	0.438198	00:23
164	1732.973877	2174.367676	2157.667725	16.617126	0.014107	0.807602	-0.483685	5860.258789	1.053869	495.876831	413.453308	0.443348	00:23
165	1735.284668	2110.953857	2095.302979	15.724959	0.004811	0.813329	-0.467449	5644.609375	1.044499	491.805328	404.473846	0.435122	00:23
166	1714.356567	2040.991577	2025.521973	15.638241	0.014943	0.812154	-0.470701	5513.529297	1.041535	489.991791	403.137115	0.435767	00:23
167	1712.174316	2073.326416	2059.069824	14.374624	0.011469	0.785766	-0.535316	5697.384277	1.003952	473.134125	384.395599	0.431473	00:24
168	1707.242554	2032.528687	2017.372681	15.232775	0.021898	0.799380	-0.502715	5552.247070	1.024798	482.139496	395.457214	0.437737	00:24
169	1701.608398	2100.842285	2086.958496	14.016817	0.014957	0.784930	-0.536719	5804.063477	0.995842	470.380188	379.255341	0.428535	00:24
170	1705.426025	2127.073975	2112.207520	14.765400	0.014636	0.791323	-0.521682	5771.546875	1.013267	477.264832	389.201874	0.432916	00:23
171	1697.164307	2029.953979	2015.340576	14.608382	0.009948	0.790753	-0.524450	5598.907227	1.012656	476.437225	389.680969	0.437537	00:24
172	1701.335815	2101.848389	2087.525146	14.297215	0.010021	0.777077	-0.558810	5761.908203	0.997443	469.637146	383.354065	0.436209	00:24
173	1687.971802	2035.941162	2021.396240	14.679951	0.011481	0.782743	-0.545734	5502.674316	1.005859	472.835815	387.943665	0.441890	00:24
174	1688.044556	2100.105225	2084.287354	15.559942	0.014354	0.788969	-0.529777	5633.252441	1.025336	482.666351	398.882202	0.441224	00:23
175	1676.035522	2010.885132	1996.957642	13.989811	0.008462	0.779063	-0.553532	5533.621094	0.994666	467.717865	380.985687	0.434618	00:23
176	1671.125122	2004.597290	1990.845215	13.638359	0.008967	0.768718	-0.580175	5492.801758	0.981971	461.985138	375.944366	0.435246	00:23
177	1665.241699	2052.365723	2037.499146	14.771413	0.009575	0.782816	-0.544561	5539.471191	1.009021	474.567261	389.439423	0.437188	00:24
178	1665.223999	2045.858154	2032.487183	13.229947	0.010469	0.767099	-0.583029	5713.245605	0.973069	458.953705	370.179657	0.429697	00:23
179	1666.685669	2031.299805	2017.070679	14.342589	0.009825	0.770655	-0.575405	5579.201172	0.990290	466.026184	380.720123	0.435823	00:24
180	1663.915894	2028.667969	2014.607666	13.977999	0.016078	0.769739	-0.577955	5492.001465	0.987753	464.716583	379.614777	0.436295	00:24
181	1655.057007	1997.265015	1982.822021	14.302199	0.011690	0.766108	-0.587889	5421.292480	0.988498	464.869781	381.413086	0.438475	00:23
182	1647.470703	2001.200928	1987.254395	13.935952	0.011662	0.765908	-0.588284	5481.558594	0.982021	461.706665	377.253204	0.438083	00:24
183	1644.264404	2038.258911	2025.146362	13.109586	0.014235	0.759005	-0.605171	5534.289551	0.964562	454.472351	367.980865	0.433600	00:24
184	1644.385620	2047.144287	2032.844604	14.502955	0.011979	0.762904	-0.596886	5548.737793	0.986703	463.879425	381.653381	0.439943	00:23
185	1636.942871	2006.265381	1991.897583	14.197330	0.010794	0.762206	-0.599666	5424.100586	0.985522	463.111511	381.617706	0.443979	00:24
186	1634.921997	1972.216187	1959.202271	13.357076	0.011709	0.746162	-0.640975	5457.113770	0.956162	449.686615	368.055725	0.439773	00:24
187	1629.582520	1984.731201	1971.263550	13.300425	0.012171	0.747330	-0.638034	5436.582031	0.958278	450.679535	369.046844	0.440862	00:24
188	1623.882446	1975.131104	1961.552612	13.839017	0.011932	0.753823	-0.620902	5386.224609	0.969848	455.789917	374.059143	0.440880	00:24
189	1627.823242	1960.349243	1947.042725	13.134436	0.012697	0.748932	-0.633633	5391.917480	0.959833	451.228424	369.377655	0.439953	00:23
190	1629.562866	1960.382568	1946.881348	13.574226	0.012456	0.750285	-0.630250	5368.029785	0.964561	453.250061	371.947876	0.440743	00:24
191	1618.520630	1960.600586	1947.427124	13.270995	0.012711	0.744906	-0.644753	5373.071777	0.957177	449.862152	369.332611	0.441584	00:24
192	1623.590454	1950.550171	1937.349487	13.391574	0.011325	0.741666	-0.653698	5364.424805	0.954702	448.554230	369.055206	0.442402	00:23
193	1614.169922	1949.909790	1936.643921	13.437939	0.011754	0.744202	-0.647078	5353.138672	0.957277	449.707184	369.869202	0.442982	00:23
194	1614.446899	1949.415161	1936.062012	13.407144	0.010088	0.741968	-0.652905	5357.693848	0.955644	449.010956	369.513000	0.442485	00:23
195	1610.449463	1948.377686	1935.371216	13.316976	0.011583	0.740351	-0.657302	5344.466797	0.951816	447.198792	367.799164	0.442478	00:23
196	1609.498535	1948.998901	1935.809692	12.996377	0.011523	0.733751	-0.675311	5363.760742	0.944965	444.183533	366.061371	0.443487	00:24
197	1607.115356	1946.807129	1933.686768	13.057825	0.011373	0.739433	-0.659943	5346.859375	0.952111	447.346741	368.376556	0.443361	00:23
198	1615.034180	1946.700073	1933.425415	13.417556	0.011178	0.741674	-0.653776	5341.712402	0.954577	448.507629	369.013855	0.442932	00:24
199	1613.031616	1946.369873	1933.160156	13.242634	0.010880	0.738820	-0.661524	5349.308594	0.951218	447.030548	368.003479	0.443090	00:24

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	64307.953125	58279.812500	58243.433594	-0.050396	0.446575	-1.720978	633.586060	0.047932	64056.988281	0.950300	0.219880	0.624841	00:24
1	52256.531250	47517.250000	47401.242188	-0.095860	0.402663	-2.017131	589.741516	0.185387	58108.828125	0.767923	0.194185	0.970138	00:23
2	34960.160156	31226.734375	30933.964844	-0.076051	0.269091	-2.800119	751.623657	0.385156	45276.707031	0.674353	0.099892	0.940919	00:24
3	24776.472656	20445.687500	20104.564453	-0.063708	0.384149	-2.067982	556.550415	0.607671	36356.878906	0.703798	0.140935	0.863217	00:23
4	17783.804688	16285.233398	15865.710938	-0.067235	0.429491	-1.891830	517.371826	0.808860	31224.277344	0.709278	0.170626	0.993864	00:24
5	13278.303711	11126.129883	10704.387695	-0.060666	0.502716	-1.599754	443.023224	0.948876	24944.644531	0.696404	0.200470	1.058257	00:24
6	10607.653320	9407.301758	8986.120117	-0.058500	0.538358	-1.487504	420.064575	0.999986	22220.373047	0.692390	0.222120	1.125344	00:24
7	9076.883789	8579.513672	8182.976074	-0.048066	0.568540	-1.401409	394.949036	1.000000	20782.205078	0.669947	0.237349	1.191771	00:24
8	8298.398438	8123.069824	7729.753418	-0.048600	0.572593	-1.406226	391.598938	1.000000	20188.673828	0.652731	0.242710	1.237344	00:24
9	7958.245605	8007.626953	7610.983887	-0.048821	0.575821	-1.404351	394.969757	1.000000	19997.000000	0.657966	0.246665	1.262074	00:24

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	7446.696289	7002.534668	6594.965820	-0.056031	0.609424	-1.249221	406.261200	1.000000	16749.544922	0.762242	0.250232	1.164957	00:24
1	6559.509766	6201.054688	5791.714844	-0.041461	0.618311	-1.230625	408.183350	1.000000	14291.185547	0.769989	0.256279	1.197002	00:24
2	5965.905762	5736.044434	5315.900879	-0.065841	0.617567	-1.259662	419.067261	1.000000	13115.260742	0.771636	0.256352	1.277502	00:24
3	5546.066406	5484.937012	5098.352539	-0.031883	0.655282	-1.149140	385.752136	1.000000	12599.239258	0.741821	0.269175	1.304524	00:23
4	5216.171387	5357.946289	4967.701172	-0.048715	0.659577	-1.167782	389.449951	1.000000	11821.793945	0.728742	0.278477	1.384185	00:24
5	4968.410156	4974.411133	4575.067383	-0.037054	0.655091	-1.190241	398.651703	1.000000	11053.794922	0.738295	0.278551	1.407455	00:24
6	4758.382812	4930.711914	4530.911133	-0.042303	0.655234	-1.206222	399.300415	1.000000	10765.223633	0.727668	0.281477	1.447791	00:24
7	4598.325684	4773.492188	4384.264648	-0.028840	0.662661	-1.158184	388.670288	1.000000	10158.075195	0.732503	0.278200	1.385599	00:24
8	4481.006348	4676.386719	4256.562988	-0.047947	0.652907	-1.256124	419.240143	1.000000	9864.458008	0.734679	0.288794	1.560645	00:24
9	4396.450195	4542.811035	4142.692871	-0.031679	0.664870	-1.194383	399.589661	1.000000	9653.433594	0.725160	0.288026	1.493910	00:23
10	4277.703125	4514.132324	4093.090332	-0.038448	0.659772	-1.263761	420.567749	1.000000	9540.075195	0.722800	0.298543	1.615734	00:24
11	4211.627441	4451.823730	4059.461670	-0.017897	0.679049	-1.174503	391.879242	1.000000	9462.656250	0.704454	0.300546	1.540585	00:23
12	4159.201660	4370.717285	3985.004150	-0.029835	0.686301	-1.132567	385.226257	1.000000	9333.199219	0.711258	0.301613	1.484894	00:24
13	4089.183594	4338.795898	3938.385986	-0.027388	0.671400	-1.209984	399.877716	1.000000	9139.356445	0.703476	0.301768	1.563450	00:23
14	4024.176514	4243.605957	3844.971924	-0.038013	0.681120	-1.182844	398.243713	1.000000	8863.547852	0.705809	0.307898	1.564476	00:24
15	3992.836426	4182.538574	3768.346436	-0.015892	0.671695	-1.240414	413.820831	1.000000	8815.115234	0.708553	0.310647	1.627596	00:24
16	3938.035645	4254.535156	3836.140381	-0.021022	0.672339	-1.260805	417.951782	1.000000	8692.565430	0.700489	0.314746	1.681814	00:24
17	3915.346924	4136.510254	3734.750488	-0.037996	0.684208	-1.182841	401.324951	1.000000	8577.833008	0.708467	0.312179	1.580640	00:24
18	3851.800781	4099.753418	3683.509521	-0.035467	0.673168	-1.264503	415.946136	1.000000	8637.051758	0.690829	0.317606	1.686912	00:24
19	3829.080811	4061.842773	3656.622803	-0.020944	0.678215	-1.224614	404.792664	1.000000	8405.791992	0.691823	0.315146	1.633282	00:24
20	3814.291260	4050.546143	3625.365479	-0.017918	0.671568	-1.263860	424.740845	1.000000	8314.105469	0.712885	0.319519	1.662932	00:24
21	3790.595947	4220.412598	3789.602051	-0.022943	0.678726	-1.280803	430.250519	1.000000	8668.383789	0.700995	0.329524	1.751208	00:23
22	3769.847412	4021.063232	3616.341309	-0.031142	0.680278	-1.221621	404.374634	1.000000	8211.235352	0.687985	0.318570	1.633209	00:24
23	3756.498779	3959.629883	3536.583008	-0.020466	0.670928	-1.292554	422.621613	1.000000	8118.128418	0.686245	0.323532	1.728216	00:24
24	3706.817871	4126.592773	3708.841553	-0.023985	0.678128	-1.259123	417.260498	1.000000	8252.971680	0.689504	0.326434	1.692541	00:24
25	3687.163330	4039.027832	3624.490723	-0.023506	0.679744	-1.252728	414.078033	1.000000	8196.382812	0.684681	0.328139	1.685571	00:24
26	3664.687012	3962.185303	3553.306396	-0.020275	0.681983	-1.241518	408.559418	1.000000	8060.037109	0.678305	0.328026	1.677929	00:24
27	3655.794189	3935.266602	3520.048096	-0.026080	0.684890	-1.246303	414.766052	1.000000	8066.496094	0.683167	0.332905	1.703669	00:24
28	3633.048096	3900.002441	3477.207520	-0.031444	0.678460	-1.280428	422.386871	1.000000	7979.427734	0.682424	0.332304	1.739592	00:23
29	3605.916504	3902.634277	3490.624512	-0.027529	0.688832	-1.218063	411.608704	1.000000	7930.282715	0.693239	0.331436	1.667827	00:23
30	3597.799072	3864.459473	3440.049072	-0.031359	0.680125	-1.279481	424.118408	1.000000	7889.754883	0.684454	0.334076	1.745801	00:23
31	3577.369385	3835.365479	3421.598145	-0.031690	0.687918	-1.243465	413.432159	1.000000	7764.869629	0.677153	0.336913	1.711155	00:23
32	3567.163574	3808.568115	3392.390869	-0.030360	0.685564	-1.245607	415.832184	1.000000	7823.952148	0.684545	0.334472	1.705133	00:24
33	3556.911133	3839.799316	3426.487305	-0.025034	0.679174	-1.259462	412.899506	1.000000	8020.717285	0.675566	0.331511	1.691069	00:23
34	3547.733643	3787.851562	3366.338867	-0.033549	0.684411	-1.252932	421.170959	1.000000	7632.813965	0.692167	0.335533	1.710420	00:23
35	3498.793457	3750.230957	3341.822998	-0.024690	0.690246	-1.227408	408.039459	1.000000	7705.300781	0.674841	0.336464	1.692907	00:24
36	3497.622314	3779.739746	3356.960205	-0.031387	0.678122	-1.298752	422.325592	1.000000	7688.250488	0.666296	0.338358	1.765067	00:23
37	3518.062500	3770.506348	3337.693848	-0.026667	0.684909	-1.278552	432.439392	1.000000	7656.469727	0.695298	0.342245	1.764647	00:24
38	3493.117676	3758.422607	3342.844238	-0.021858	0.688443	-1.241507	415.205994	1.000000	7605.489746	0.681559	0.339791	1.703853	00:24
39	3465.591064	3732.348145	3307.750488	-0.019985	0.681497	-1.268380	424.199097	1.000000	7533.059082	0.689752	0.338435	1.715226	00:24
40	3459.664551	3786.777588	3376.685059	-0.013549	0.681292	-1.238933	409.652313	1.000000	7627.532227	0.682110	0.330924	1.658110	00:23
41	3466.934326	3731.796387	3311.013428	-0.029070	0.678285	-1.272447	420.435974	1.000000	7626.694824	0.682827	0.335078	1.704524	00:23
42	3442.351074	3738.564697	3302.626465	-0.041683	0.680691	-1.311230	435.543732	1.000000	7666.170898	0.680946	0.344484	1.805883	00:24
43	3429.302246	3697.198730	3261.440674	-0.023874	0.680301	-1.309720	435.364044	1.000000	7565.525879	0.682782	0.345298	1.794736	00:24
44	3421.857422	3716.597656	3288.889404	-0.018520	0.678248	-1.302927	427.315948	1.000000	7654.982910	0.672833	0.342392	1.766432	00:23
45	3432.586670	3763.864990	3343.723389	-0.013860	0.682215	-1.270576	419.691833	1.000000	7661.630859	0.677036	0.339797	1.728749	00:24
46	3409.428711	3733.609619	3305.357666	-0.026532	0.675712	-1.306996	427.861542	1.000000	7548.878906	0.673747	0.341761	1.756524	00:24
47	3396.132812	3671.088623	3244.154297	-0.027941	0.680358	-1.281522	426.605988	1.000000	7443.862793	0.685382	0.340985	1.736874	00:24
48	3388.988281	3633.996582	3204.664795	-0.028510	0.683743	-1.284135	428.964844	1.000000	7269.158203	0.683487	0.344635	1.762939	00:23
49	3374.890625	3655.472168	3228.736816	-0.025874	0.681976	-1.279506	426.387177	1.000000	7347.823730	0.683649	0.342960	1.738811	00:24
50	3374.364014	3661.433594	3226.835205	-0.027927	0.680691	-1.301747	434.195587	1.000000	7426.177734	0.685498	0.345380	1.777534	00:24
51	3377.024414	3646.464844	3221.161377	-0.040788	0.683618	-1.276678	424.889130	1.000000	7338.713379	0.679540	0.343833	1.744919	00:24
52	3367.656982	3639.472168	3208.727295	-0.026861	0.677835	-1.289724	430.378632	1.000000	7426.609863	0.690034	0.342502	1.726464	00:24
53	3353.731689	3598.407715	3163.959717	-0.032011	0.680891	-1.294131	434.070862	1.000000	7290.533203	0.690069	0.345623	1.758445	00:24
54	3350.532715	3618.042969	3183.317627	-0.024679	0.679061	-1.301106	434.440765	1.000000	7441.650879	0.688150	0.345228	1.761177	00:24
55	3328.891113	3591.654785	3151.659912	-0.024667	0.675619	-1.320078	439.684174	1.000000	7220.856445	0.689841	0.345016	1.782546	00:24
56	3320.435059	3628.756104	3184.838135	-0.023147	0.673282	-1.327310	443.469299	1.000000	7381.559082	0.695388	0.344838	1.779874	00:24
57	3319.204590	3589.929688	3161.705322	-0.029322	0.681353	-1.283165	427.894348	1.000000	7210.187988	0.684854	0.343528	1.745058	00:24
58	3333.072510	3605.556396	3159.020752	-0.022901	0.669782	-1.351565	446.191711	1.000000	7292.311523	0.687392	0.345130	1.809493	00:24
59	3301.860840	3583.256592	3147.702881	-0.018526	0.682041	-1.308480	435.160400	1.000000	7301.224609	0.678834	0.351069	1.792501	00:23
60	3289.162598	3571.066650	3126.065918	-0.025570	0.675508	-1.345270	444.585358	1.000000	7098.305664	0.679848	0.351047	1.829405	00:23
61	3285.661377	3577.617432	3150.016113	-0.022347	0.678402	-1.289324	427.211182	1.000000	7195.916504	0.681604	0.343973	1.729821	00:24
62	3292.790527	3545.149658	3106.085693	-0.020378	0.672821	-1.324593	438.766663	1.000000	7157.503418	0.687220	0.345001	1.767740	00:24
63	3273.931641	3576.871826	3127.184814	-0.030022	0.663642	-1.371861	449.397675	1.000000	7171.856934	0.685767	0.344284	1.808998	00:24
64	3286.415527	3542.644287	3110.177246	-0.025038	0.680133	-1.290051	432.100250	1.000000	7128.390137	0.689775	0.345855	1.743150	00:24
65	3295.292969	3554.997070	3110.141602	-0.031996	0.673540	-1.316412	444.468079	1.000000	7076.810547	0.704291	0.345098	1.755248	00:24
66	3287.177979	3522.089844	3080.620850	-0.039147	0.673940	-1.317083	441.162537	1.000000	7100.459961	0.695908	0.345708	1.758326	00:24
67	3241.864502	3488.717773	3024.696045	-0.025456	0.659230	-1.414685	463.702454	1.000000	7130.245605	0.688320	0.351045	1.855794	00:23
68	3192.238281	3457.201172	3000.528809	-0.025450	0.664398	-1.374820	456.514099	1.000000	6979.332031	0.696865	0.349181	1.810351	00:24
69	3159.333496	3475.097168	3000.846436	-0.023379	0.655233	-1.422044	474.188934	1.000000	7015.226074	0.712399	0.347041	1.856408	00:23
70	3117.958740	3506.858643	3045.154053	-0.011474	0.658447	-1.394947	461.412323	1.000000	7187.719727	0.700233	0.347915	1.812438	00:24
71	3089.015869	3462.144531	3013.021973	-0.024864	0.666162	-1.344185	448.933685	1.000000	7037.438965	0.701362	0.347054	1.756467	00:23
72	3063.737793	3324.433838	2869.878662	-0.035118	0.660016	-1.366300	454.371368	1.000000	6876.746582	0.703546	0.345587	1.760979	00:24
73	3027.214111	3331.492920	2870.182617	-0.011670	0.660568	-1.397654	461.116272	1.000000	6893.470215	0.693894	0.351798	1.828764	00:24
74	3017.817139	3321.054199	2865.548340	-0.025422	0.660219	-1.382914	455.403198	1.000000	6842.084473	0.691309	0.352129	1.784368	00:24
75	2992.729736	3256.467041	2794.733398	-0.019993	0.661402	-1.395585	461.590149	1.000000	6778.131348	0.693400	0.355003	1.819794	00:23
76	2983.455566	3289.244873	2831.101807	-0.029130	0.667984	-1.363975	457.947998	1.000000	6734.300781	0.701110	0.355252	1.800379	00:23
77	2976.864746	3239.218018	2781.242432	-0.028557	0.660307	-1.386050	457.947540	1.000000	6727.701172	0.694157	0.352464	1.793653	00:23
78	2956.301025	3304.631104	2842.858887	-0.027479	0.657182	-1.408286	461.591095	1.000000	6794.987793	0.687993	0.355320	1.808959	00:24
79	2945.924316	3334.847900	2866.822021	-0.024711	0.658704	-1.412751	467.761139	1.000000	6867.231934	0.696831	0.356441	1.834160	00:23
80	2924.367188	3198.714355	2731.728760	-0.018036	0.659534	-1.423019	466.900421	1.000000	6670.595703	0.685548	0.359238	1.858049	00:24
81	2906.989014	3215.365967	2753.775635	-0.021889	0.660180	-1.399861	461.448822	1.000000	6624.345215	0.690112	0.357600	1.810584	00:24
82	2896.910645	3215.521729	2746.866455	-0.023133	0.657917	-1.429965	468.538452	1.000000	6574.671387	0.685399	0.359657	1.859299	00:24
83	2868.951416	3202.712891	2736.927490	-0.030787	0.663033	-1.420170	465.648895	1.000000	6593.380371	0.678296	0.363546	1.870697	00:23
84	2858.278809	3162.275635	2696.020264	-0.019797	0.660603	-1.421584	466.094910	1.000000	6575.957520	0.682723	0.360896	1.858423	00:24
85	2846.207275	3103.525635	2629.209473	-0.024866	0.656847	-1.462483	474.175415	1.000000	6484.671875	0.671980	0.365991	1.908332	00:24
86	2817.036377	3078.863525	2616.457520	-0.022164	0.662941	-1.424628	462.297211	1.000000	6458.416992	0.666711	0.365627	1.872652	00:24
87	2798.375732	3076.187744	2600.413330	-0.032358	0.658778	-1.468475	475.640625	1.000000	6369.106445	0.666486	0.369961	1.928813	00:24
88	2771.174316	3070.136230	2606.374756	-0.024365	0.665567	-1.433004	463.692261	1.000000	6426.307617	0.658267	0.370472	1.902251	00:24
89	2762.876221	3072.244873	2603.819824	-0.026865	0.662920	-1.453525	468.355865	1.000000	6418.618652	0.655338	0.372392	1.920821	00:23
90	2738.615723	3011.625732	2538.651123	-0.022828	0.662585	-1.463570	472.897278	1.000000	6366.740723	0.658284	0.373876	1.937470	00:23
91	2722.517822	2995.657715	2518.491211	-0.018094	0.661352	-1.488024	477.079224	1.000000	6295.547852	0.649196	0.376828	1.975732	00:24
92	2708.709961	3014.197021	2543.201416	-0.019909	0.665528	-1.464587	470.887909	1.000000	6312.110840	0.647752	0.377463	1.954773	00:24
93	2697.568115	2966.013672	2492.254150	-0.020296	0.664736	-1.472951	473.693115	1.000000	6222.543945	0.648153	0.378263	1.965246	00:23
94	2692.429199	2959.528076	2484.598633	-0.019818	0.663212	-1.485151	474.830475	1.000000	6250.688965	0.642955	0.378584	1.979259	00:23
95	2679.370361	2948.991699	2474.185791	-0.017727	0.664275	-1.484554	474.726776	1.000000	6253.794922	0.641554	0.379589	1.984106	00:24
96	2667.490234	2942.266357	2468.315674	-0.019934	0.664913	-1.478830	473.874390	1.000000	6198.797852	0.643243	0.379516	1.976109	00:24
97	2663.017822	2939.738770	2463.631104	-0.018928	0.664092	-1.489275	476.029968	1.000000	6207.451660	0.640717	0.380436	1.990803	00:24
98	2661.197754	2935.853516	2459.528320	-0.019907	0.664481	-1.487758	476.260712	1.000000	6187.456543	0.641910	0.380615	1.990031	00:24
99	2662.050293	2935.732422	2458.348145	-0.019693	0.665127	-1.483711	477.305267	1.000000	6187.085449	0.647012	0.380451	1.986876	00:24

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	57119.871094	48978.390625	48411.449219	-0.171113	0.467323	-1.606784	11681.308594	0.047932	57218.921875	4.143692	0.132895	0.589939	00:25
1	42990.976562	33256.042969	32888.765625	-0.111939	0.787082	-0.534524	1944.819458	0.185387	45751.316406	1.667504	0.184117	0.476903	00:25
2	27649.550781	18003.472656	17567.388672	-0.051287	0.614378	-1.090928	1118.868530	0.385156	32650.421875	1.105906	0.211503	0.686238	00:25
3	18592.869141	13246.469727	12702.623047	-0.040173	0.588392	-1.146373	889.270142	0.607671	27025.304688	0.926397	0.170838	0.591538	00:25
4	13824.781250	11302.725586	10674.322266	-0.025407	0.626357	-1.017098	773.015381	0.808860	24171.271484	0.872669	0.170972	0.585883	00:25
5	11246.221680	9732.048828	9055.437500	0.002667	0.623677	-1.060219	710.477600	0.948876	21821.728516	0.787523	0.195109	0.716930	00:25
6	9790.408203	9052.549805	8347.827148	0.037016	0.624793	-1.088983	702.386963	0.999986	20620.949219	0.755349	0.210199	0.831470	00:25
7	8893.800781	8546.672852	7901.442383	0.032866	0.649115	-1.026540	642.988525	1.000000	19937.876953	0.710113	0.222703	0.880089	00:25
8	8421.801758	8350.280273	7711.315918	0.032067	0.647427	-1.042604	636.768677	1.000000	19623.410156	0.692194	0.226795	0.915346	00:25
9	8228.472656	8303.927734	7656.038574	0.032340	0.642907	-1.063508	645.732849	1.000000	19511.349609	0.690796	0.228713	0.935445	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	7636.996094	7720.530273	7083.126953	0.019008	0.683685	-0.985189	635.105652	1.000000	17760.312500	0.687582	0.255679	1.082731	00:26
1	6885.542480	6894.079590	6297.856445	0.006627	0.707549	-0.936249	594.373535	1.000000	16222.292969	0.647742	0.266870	1.149313	00:25
2	6309.057617	6247.788574	5663.020508	0.013977	0.733179	-0.899275	583.153992	1.000000	14961.074219	0.627324	0.282289	1.256484	00:25
3	5862.837891	5824.015137	5272.883301	0.018640	0.760227	-0.806429	549.852478	1.000000	13915.848633	0.627636	0.282030	1.206542	00:25
4	5524.013672	5580.453613	4991.576660	0.000202	0.744082	-0.900245	587.568359	1.000000	13173.700195	0.615521	0.290766	1.348567	00:25
5	5247.176758	5352.265625	4772.904785	0.012758	0.746673	-0.887565	578.224121	1.000000	12680.643555	0.611426	0.288877	1.332583	00:25
6	5000.711914	5149.272949	4567.143555	0.021369	0.757468	-0.872349	581.102356	1.000000	12148.317383	0.609617	0.297950	1.367787	00:25
7	4822.857422	4957.025879	4346.383789	0.012121	0.743142	-0.944743	609.705933	1.000000	11592.447266	0.603416	0.300536	1.456099	00:25
8	4670.200195	4811.929199	4201.861328	0.023626	0.744356	-0.943792	609.286682	1.000000	11109.132812	0.600485	0.303331	1.457995	00:25
9	4552.351562	4701.762207	4124.187012	0.012166	0.752433	-0.885569	576.840942	1.000000	10700.002930	0.601379	0.294862	1.367969	00:25
10	4457.607422	4782.782715	4139.709961	0.030688	0.750395	-0.973056	642.217896	1.000000	10746.132812	0.605009	0.315268	1.573785	00:25
11	4389.464844	4600.456543	3989.061279	0.006628	0.754086	-0.906970	610.690735	1.000000	10181.538086	0.620296	0.304632	1.429988	00:25
12	4316.668945	4546.292480	3929.669189	0.008763	0.755007	-0.928994	615.957153	1.000000	10081.615234	0.603556	0.313128	1.485501	00:26
13	4249.416016	4501.651855	3910.774902	0.001233	0.753363	-0.902888	590.159973	1.000000	9931.499023	0.598335	0.304420	1.410765	00:25
14	4194.772461	4419.601562	3797.657227	0.011631	0.748259	-0.947026	621.403442	1.000000	9608.806641	0.604558	0.311978	1.482321	00:26
15	4155.236328	4390.953125	3742.381348	0.013369	0.749580	-0.965407	648.057068	1.000000	9297.745117	0.618429	0.318911	1.533816	00:25
16	4107.301758	4357.127441	3726.751709	0.014419	0.751795	-0.950445	629.798523	1.000000	9370.619141	0.605639	0.317125	1.514558	00:25
17	4080.357178	4326.577148	3693.467285	0.019456	0.757275	-0.930076	632.673401	1.000000	9206.707031	0.618530	0.317690	1.501742	00:25
18	4047.259766	4325.830566	3670.197754	0.016036	0.747400	-0.985562	655.035889	1.000000	9117.528320	0.611593	0.321978	1.569723	00:25
19	4017.182617	4268.327637	3646.175049	0.016499	0.751013	-0.935652	621.611816	1.000000	9072.065430	0.609426	0.314141	1.470098	00:25
20	3994.158936	4206.021484	3567.909668	0.020197	0.751552	-0.950140	637.651001	1.000000	8851.089844	0.615019	0.318669	1.509441	00:25
21	3978.268555	4256.660156	3621.489990	0.006194	0.745934	-0.969727	634.768066	1.000000	8992.529297	0.602167	0.318927	1.514271	00:26
22	3955.441650	4265.469238	3629.651123	-0.002801	0.752090	-0.953087	635.321167	1.000000	8820.404297	0.607925	0.321245	1.517332	00:26
23	3929.008545	4206.423828	3560.397217	0.014343	0.746978	-0.970275	645.613831	1.000000	8713.512695	0.614093	0.319624	1.526678	00:25
24	3908.563965	4153.493164	3523.141113	0.006290	0.751558	-0.946957	629.941040	1.000000	8658.349609	0.608749	0.317679	1.503032	00:25
25	3898.939697	4141.995605	3511.120850	0.015062	0.746144	-0.961228	630.455627	1.000000	8498.142578	0.604333	0.316788	1.497967	00:25
26	3885.578369	4103.011230	3454.336182	0.016320	0.742984	-0.985140	648.239014	1.000000	8472.931641	0.609011	0.321561	1.529650	00:25
27	3885.664307	4144.894043	3514.354492	0.007924	0.749983	-0.937831	630.100952	1.000000	8521.073242	0.619199	0.315660	1.465887	00:25
28	3863.140381	4171.204590	3528.229492	0.008750	0.749470	-0.969313	642.520569	1.000000	8679.597656	0.605489	0.324102	1.536831	00:25
29	3844.579834	4065.047852	3432.645752	0.014242	0.751757	-0.957529	631.998291	1.000000	8357.607422	0.599753	0.322359	1.528431	00:25
30	3806.262695	4071.528809	3425.390625	0.006845	0.749838	-0.974802	645.797546	1.000000	8272.665039	0.603490	0.326274	1.551809	00:25
31	3803.645508	4030.028564	3395.478516	0.018196	0.753908	-0.944004	634.221741	1.000000	8226.596680	0.611156	0.321838	1.508687	00:25
32	3798.153320	4027.861572	3400.078369	0.002770	0.757909	-0.921416	627.355713	1.000000	8197.963867	0.616403	0.321067	1.476825	00:25
33	3795.605469	4068.377686	3419.073730	-0.000919	0.747791	-0.983250	648.834473	1.000000	8331.038086	0.603142	0.326154	1.559857	00:26
34	3774.649170	4009.320557	3368.954346	0.006577	0.745278	-0.975523	640.058167	1.000000	8151.725098	0.603200	0.322254	1.523311	00:25
35	3797.206055	4070.147705	3423.396484	0.010856	0.747066	-0.977548	646.342896	1.000000	8184.187500	0.606127	0.324828	1.541001	00:25
36	3764.414795	3997.911377	3347.640381	0.000495	0.740897	-0.989360	649.856018	1.000000	8078.457031	0.609805	0.322489	1.524083	00:25
37	3738.304443	4000.742188	3355.457764	0.006559	0.746498	-0.979872	644.819946	1.000000	8064.453125	0.602224	0.326189	1.537351	00:25
38	3721.401367	4012.916016	3346.493408	0.014446	0.741339	-0.996954	665.997559	1.000000	7987.633789	0.622092	0.324676	1.547024	00:25
39	3733.728516	4002.178223	3343.464111	0.003242	0.744426	-0.993874	658.359314	1.000000	8037.950195	0.610395	0.326803	1.561654	00:25
40	3713.223877	3972.769287	3311.979980	0.027494	0.745946	-0.984094	660.406372	1.000000	7864.602051	0.618499	0.326929	1.545728	00:25
41	3725.438721	4000.221680	3349.538574	0.010210	0.749517	-0.965508	650.194214	1.000000	7884.955078	0.617907	0.324975	1.529040	00:25
42	3714.680908	4006.921143	3337.395020	0.006614	0.741850	-1.008360	669.165833	1.000000	7901.441406	0.614372	0.328160	1.577131	00:25
43	3684.118408	3953.133789	3283.846680	0.002745	0.745630	-1.007263	668.936707	1.000000	7932.915527	0.609230	0.331397	1.602322	00:25
44	3617.731934	3961.301758	3277.286377	0.008253	0.732352	-1.054439	683.772156	1.000000	8140.794434	0.605360	0.331876	1.616396	00:25
45	3527.310547	3771.926025	3082.391357	0.021190	0.732152	-1.045097	689.335327	1.000000	7549.060059	0.621409	0.328936	1.597990	00:25
46	3481.053223	3792.558594	3111.916016	0.014146	0.736934	-1.022333	680.446411	1.000000	7618.551758	0.623152	0.328771	1.574195	00:25
47	3449.490723	3723.544434	3044.811035	0.004227	0.736182	-1.023490	678.525024	1.000000	7445.786133	0.619757	0.330939	1.563343	00:26
48	3453.458984	3735.135986	3037.158936	0.002894	0.727846	-1.065144	697.809265	1.000000	7540.457520	0.619377	0.332484	1.605249	00:25
49	3440.742188	3698.924316	3023.513428	0.015087	0.739747	-1.005747	675.299194	1.000000	7419.930664	0.626200	0.329499	1.548569	00:25
50	3408.526855	3675.649902	3002.518066	0.007204	0.733653	-1.024227	672.936035	1.000000	7433.500000	0.616485	0.328433	1.547451	00:25
51	3414.460693	3745.192383	3054.504639	0.023048	0.737199	-1.023411	690.448914	1.000000	7412.087402	0.632781	0.331396	1.572160	00:25
52	3390.644287	3662.520020	2988.273438	0.007120	0.741236	-1.006877	674.001587	1.000000	7432.633301	0.620694	0.332220	1.557971	00:25
53	3382.684326	3674.317383	3003.882568	-0.000215	0.739698	-1.012987	670.337097	1.000000	7319.799805	0.612742	0.332873	1.558902	00:25
54	3380.493896	3745.933105	3057.020508	0.010614	0.740175	-1.024149	688.698853	1.000000	7467.467773	0.624081	0.336852	1.585891	00:25
55	3365.373047	3633.520752	2938.203613	0.000266	0.735362	-1.048191	695.175659	1.000000	7304.637695	0.618786	0.336548	1.616588	00:25
56	3360.692871	3634.707031	2946.971191	0.006276	0.732114	-1.034212	687.640747	1.000000	7217.326172	0.627182	0.330707	1.557482	00:25
57	3361.901367	3677.013184	2977.446289	0.008722	0.729319	-1.064693	699.587585	1.000000	7371.806152	0.619002	0.334934	1.612427	00:26
58	3341.074951	3619.520508	2932.792236	0.004525	0.736755	-1.030560	686.562866	1.000000	7233.354980	0.621963	0.333719	1.583878	00:25
59	3341.017334	3677.554688	2991.288574	-0.001353	0.736229	-1.028123	686.075989	1.000000	7211.670410	0.624420	0.332879	1.574305	00:25
60	3336.556885	3614.616699	2914.052734	0.012258	0.729867	-1.065539	700.479431	1.000000	7202.227539	0.618801	0.334756	1.621663	00:26
61	3338.885986	3666.255859	2975.747559	0.011388	0.735329	-1.025145	690.370422	1.000000	7230.252930	0.633762	0.331731	1.559722	00:25
62	3316.322510	3614.233643	2910.035645	0.011739	0.730760	-1.060256	703.997192	1.000000	7130.198242	0.625956	0.335665	1.610752	00:25
63	3306.802246	3591.011963	2898.936035	0.019578	0.729113	-1.051367	692.025085	1.000000	7151.365234	0.622282	0.331209	1.580528	00:25
64	3319.408691	3607.750244	2936.093506	0.015589	0.738210	-1.010526	671.561401	1.000000	7256.174805	0.619674	0.329484	1.550395	00:25
65	3329.283447	3666.196533	2997.034180	0.007939	0.738448	-1.005233	669.044373	1.000000	7284.672363	0.620451	0.329718	1.533679	00:25
66	3319.029053	3617.955322	2919.307373	0.009864	0.730675	-1.044238	698.430969	1.000000	7143.467285	0.634253	0.331163	1.574749	00:25
67	3304.009033	3641.074951	2941.493896	0.036581	0.734014	-1.039964	699.558655	1.000000	7150.225098	0.633657	0.332079	1.591718	00:25
68	3283.734131	3604.605469	2921.168701	0.009594	0.734629	-1.032366	683.271362	1.000000	7177.411621	0.619841	0.331710	1.576211	00:25
69	3282.202637	3547.863525	2858.945801	0.018959	0.732195	-1.035658	688.809265	1.000000	7019.934570	0.627719	0.329841	1.567864	00:26
70	3285.655762	3639.036133	2936.884277	0.013103	0.730776	-1.055586	701.959717	1.000000	7140.610840	0.627760	0.334509	1.601171	00:26
71	3288.370117	3591.168457	2893.040771	0.015652	0.726576	-1.061164	697.990662	1.000000	7123.018555	0.624936	0.331148	1.586430	00:25
72	3270.113281	3567.283447	2889.191162	0.009472	0.737333	-1.018427	677.971008	1.000000	7061.854004	0.621693	0.330927	1.562181	00:25
73	3268.113281	3598.905273	2899.352539	0.011853	0.725724	-1.065427	699.484924	1.000000	7144.051270	0.624485	0.331081	1.592555	00:25
74	3265.667236	3563.320068	2867.907227	0.012042	0.729498	-1.047444	695.306763	1.000000	7025.574219	0.628830	0.331620	1.572838	00:25
75	3245.814453	3517.445068	2836.110107	0.003315	0.734034	-1.036802	681.251587	1.000000	7037.501953	0.613656	0.333098	1.580311	00:25
76	3241.935303	3530.926758	2817.048340	0.011109	0.723914	-1.084425	713.788513	1.000000	6924.078125	0.627887	0.333674	1.624286	00:25
77	3245.354980	3546.132568	2842.161621	0.005974	0.725738	-1.074948	703.946472	1.000000	6994.406738	0.621140	0.333325	1.616471	00:25
78	3245.945557	3528.050781	2828.606201	0.012258	0.728167	-1.062054	699.372986	1.000000	6909.879883	0.623076	0.333196	1.599359	00:25
79	3229.143799	3506.233887	2806.143555	0.004439	0.724063	-1.080432	699.957153	1.000000	6967.483398	0.614117	0.332865	1.616866	00:25
80	3220.140869	3577.505371	2893.383057	0.005646	0.731227	-1.039378	683.952393	1.000000	6991.556152	0.620166	0.329861	1.570222	00:25
81	3208.395508	3488.423340	2788.742188	-0.000503	0.729512	-1.061916	699.554688	1.000000	6825.243652	0.620982	0.335146	1.605686	00:25
82	3191.760254	3466.438232	2781.985352	0.004644	0.736124	-1.030661	684.366333	1.000000	6905.499023	0.620341	0.332895	1.582907	00:25
83	3171.593750	3468.174316	2757.281982	0.013745	0.726387	-1.079011	710.753906	1.000000	6798.083984	0.624436	0.335499	1.625272	00:26
84	3161.748779	3484.867432	2804.370850	0.000396	0.730777	-1.043842	680.357971	1.000000	6938.908203	0.611897	0.331202	1.576850	00:25
85	3145.614746	3414.764160	2711.646484	0.007162	0.726800	-1.078855	703.033142	1.000000	6748.033203	0.614373	0.335700	1.630140	00:25
86	3135.015625	3464.562256	2775.013672	0.002118	0.727726	-1.071037	689.401733	1.000000	6989.595215	0.602924	0.334548	1.618078	00:26
87	3112.888184	3406.750488	2711.639893	0.013885	0.731807	-1.061808	695.017822	1.000000	6812.809570	0.611747	0.336112	1.625439	00:25
88	3104.485596	3372.153564	2665.243896	0.003058	0.727263	-1.091322	706.815735	1.000000	6685.791992	0.606084	0.340014	1.657788	00:25
89	3089.446289	3366.230469	2659.792236	0.007324	0.727721	-1.095376	706.328125	1.000000	6742.983887	0.601475	0.340201	1.673326	00:25
90	3073.190186	3344.774414	2630.106934	0.006065	0.724318	-1.111660	714.586670	1.000000	6634.008789	0.602436	0.340756	1.686546	00:25
91	3060.273926	3337.490479	2629.539795	0.006006	0.727197	-1.101563	707.863281	1.000000	6658.198730	0.598446	0.340994	1.685095	00:26
92	3047.108398	3325.574951	2611.993408	0.016005	0.727472	-1.106509	713.503357	1.000000	6600.878906	0.600146	0.342724	1.696152	00:26
93	3038.662354	3313.505615	2604.016357	0.005596	0.728518	-1.106411	709.403442	1.000000	6611.263672	0.593522	0.343257	1.704652	00:26
94	3017.475830	3303.448486	2587.911377	0.006252	0.727385	-1.113257	715.440002	1.000000	6552.413086	0.596630	0.343977	1.711420	00:25
95	3021.622803	3296.240723	2581.996338	0.005719	0.725646	-1.115477	714.178345	1.000000	6569.918457	0.595726	0.343124	1.704090	00:25
96	3009.111816	3291.238281	2580.051758	0.004037	0.726861	-1.111897	711.094604	1.000000	6562.848633	0.592960	0.343621	1.703392	00:25
97	3000.961182	3284.883545	2570.526123	0.008544	0.726318	-1.115299	714.283325	1.000000	6541.076172	0.594613	0.344068	1.706477	00:25
98	2997.922852	3285.489746	2576.586670	0.007900	0.727089	-1.110541	708.791443	1.000000	6581.392090	0.590692	0.343674	1.701313	00:25
99	2992.362305	3283.334229	2569.966553	0.007610	0.726215	-1.116118	713.273376	1.000000	6540.073730	0.592672	0.344181	1.707559	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	62175.605469	58420.937500	58348.890625	-0.051949	0.756474	-0.705867	1368.726318	0.047932	61588.308594	1.822992	0.582341	0.659328	00:25
1	49618.242188	39473.527344	39206.046875	-0.043057	0.851662	-0.396860	1403.989502	0.185387	52113.386719	2.003324	0.221583	0.661778	00:25
2	38106.187500	29982.267578	29626.966797	-0.051806	0.686124	-1.142172	902.552917	0.385156	45277.328125	1.447544	0.294982	2.087242	00:25
3	31850.349609	25701.712891	25034.757812	-0.007050	0.893151	-0.651888	1084.826294	0.607671	40963.800781	1.650774	0.426548	1.889337	00:25
4	26937.095703	22336.453125	21880.179688	-0.020213	0.755051	-0.904856	554.679199	0.808860	37525.218750	1.066666	0.338944	1.718830	00:25
5	21971.119141	18084.140625	17574.322266	0.022983	0.661450	-1.064209	531.471375	0.948876	32476.261719	1.040300	0.269217	1.212421	00:25
6	17252.421875	14166.872070	13769.662109	-0.004297	0.650027	-1.045775	395.191437	0.999986	27589.769531	0.842583	0.236650	1.021509	00:25
7	13777.228516	12127.618164	11719.044922	-0.000522	0.590726	-1.253758	407.167450	1.000000	24565.738281	0.784295	0.232350	1.025303	00:25
8	12025.313477	11425.287109	11027.707031	-0.002325	0.577320	-1.306436	396.206085	1.000000	23534.054688	0.737082	0.233305	1.023820	00:25
9	11348.297852	11215.250977	10807.715820	-0.002401	0.565716	-1.354919	405.828369	1.000000	23167.085938	0.733784	0.232911	1.041644	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	9465.326172	9999.078125	9601.225586	0.007130	0.609639	-1.205733	396.314056	1.000000	21853.699219	0.774112	0.246161	1.020246	00:26
1	8063.734375	7810.102051	7418.045410	-0.010892	0.615129	-1.217685	390.613068	1.000000	17732.515625	0.748079	0.253269	1.108232	00:25
2	7098.869141	6891.984375	6494.851074	0.003510	0.620044	-1.240355	395.655640	1.000000	15703.275391	0.735949	0.262632	1.223647	00:25
3	6485.855469	6270.389648	5868.511719	-0.005693	0.648472	-1.133566	400.792297	1.000000	13933.801758	0.791065	0.267172	1.181391	00:25
4	5972.832520	5844.192871	5451.658691	0.012890	0.644426	-1.175375	391.558289	1.000000	13166.174805	0.748259	0.268520	1.273546	00:25
5	5572.884766	5628.988281	5244.474121	0.007183	0.655677	-1.138401	383.767944	1.000000	12375.596680	0.746120	0.270278	1.270705	00:25
6	5250.625977	5428.687500	5025.041504	0.000618	0.657387	-1.176482	402.747681	1.000000	12075.447266	0.754705	0.281650	1.383355	00:25
7	4992.806152	5028.703125	4640.528320	0.009451	0.677483	-1.110796	387.616364	1.000000	11064.332031	0.747032	0.286410	1.379216	00:25
8	4797.666016	4977.150879	4576.464844	-0.004673	0.662541	-1.180220	399.917969	1.000000	10845.199219	0.740392	0.284689	1.444876	00:25
9	4619.204590	4714.991211	4327.709961	-0.004181	0.676296	-1.131910	386.741913	1.000000	10164.172852	0.729932	0.291930	1.414375	00:25
10	4479.482422	4887.600586	4481.559082	0.000063	0.672485	-1.165117	405.433441	1.000000	10460.187500	0.749355	0.294086	1.469483	00:25
11	4343.376953	4530.287598	4109.966309	0.000223	0.664114	-1.229507	419.752380	1.000000	9527.458008	0.741199	0.301180	1.557173	00:25
12	4236.599609	4525.162598	4090.072266	-0.011065	0.662348	-1.276214	434.732178	1.000000	9375.357422	0.739744	0.307770	1.653390	00:25
13	4161.453613	4408.749512	3994.785400	0.002932	0.679915	-1.200745	413.446655	1.000000	9209.034180	0.729387	0.309570	1.608523	00:26
14	4097.939453	4343.777832	3945.017334	-0.005056	0.684136	-1.152501	398.164246	1.000000	8986.760742	0.727446	0.305452	1.514880	00:25
15	4039.005859	4262.605957	3840.835449	-0.013311	0.676826	-1.218664	421.277039	1.000000	8903.778320	0.734644	0.313449	1.612776	00:25
16	3961.631104	4279.156250	3818.602783	-0.023979	0.654508	-1.387810	460.055573	1.000000	8851.189453	0.717633	0.323141	1.834598	00:25
17	3883.475586	4122.465820	3724.420898	-0.008015	0.687322	-1.149417	397.567566	1.000000	8689.211914	0.723197	0.310432	1.518600	00:25
18	3840.566650	4119.035156	3691.478027	-0.026009	0.669685	-1.277979	427.093781	1.000000	8522.090820	0.709724	0.320246	1.689855	00:25
19	3812.471191	4134.383301	3698.563232	-0.014498	0.668147	-1.309822	435.315338	1.000000	8758.106445	0.705087	0.324950	1.747292	00:25
20	3767.977051	3992.742676	3547.468750	-0.034419	0.659538	-1.350781	444.878937	1.000000	8189.853516	0.704693	0.324789	1.773977	00:25
21	3723.281494	3939.358887	3537.192627	-0.013502	0.689966	-1.189019	401.773529	1.000000	8170.947266	0.695311	0.325168	1.619005	00:25
22	3688.287109	4007.643555	3580.424805	-0.018447	0.674222	-1.281021	426.764069	1.000000	8233.244141	0.699469	0.327571	1.716607	00:26
23	3678.601562	3923.854248	3505.163086	-0.016415	0.674986	-1.260992	418.299408	1.000000	8065.615723	0.694784	0.326394	1.671506	00:25
24	3630.250244	3908.575195	3461.221436	-0.023672	0.666459	-1.361809	446.985535	1.000000	7965.235840	0.690277	0.335831	1.830896	00:25
25	3626.498291	3903.759521	3496.674316	-0.013907	0.686217	-1.209509	406.692993	1.000000	8029.478516	0.693128	0.329922	1.624238	00:25
26	3597.376465	3885.768066	3456.359863	-0.008869	0.673719	-1.306522	428.977020	1.000000	7959.312012	0.683002	0.335279	1.756485	00:25
27	3593.810059	3988.692871	3541.672119	-0.011080	0.669500	-1.339139	446.569672	1.000000	8099.977051	0.701862	0.337833	1.797264	00:25
28	3573.325195	3829.908447	3395.960693	-0.011860	0.675189	-1.295936	433.591003	1.000000	7721.243164	0.699376	0.335587	1.742384	00:26
29	3564.660889	3786.928711	3345.248535	-0.024480	0.668474	-1.339564	441.403625	1.000000	7804.659668	0.690355	0.338523	1.781669	00:25
30	3538.487793	3791.620361	3366.694336	-0.016601	0.676776	-1.271384	424.608124	1.000000	7792.783691	0.696108	0.333885	1.694794	00:25
31	3548.714844	3835.811523	3416.373779	-0.026706	0.682827	-1.228750	419.062561	1.000000	7792.014648	0.708988	0.331578	1.643911	00:25
32	3513.774902	3761.795898	3319.690430	-0.018128	0.669407	-1.317629	441.705902	1.000000	7464.705566	0.707018	0.336299	1.741212	00:25
33	3502.417969	3753.980469	3323.896729	-0.011950	0.676053	-1.294433	429.727722	1.000000	7533.951172	0.689708	0.338480	1.737193	00:26
34	3477.667969	3709.925049	3269.001709	-0.004544	0.668855	-1.335782	440.547546	1.000000	7495.550781	0.690255	0.340251	1.772696	00:25
35	3460.046631	3735.198975	3312.612793	-0.010980	0.679006	-1.271484	422.251831	1.000000	7543.686035	0.685776	0.339557	1.700740	00:25
36	3457.488770	3756.474609	3301.977539	-0.015253	0.663251	-1.366208	454.068909	1.000000	7613.058105	0.702964	0.341805	1.795775	00:25
37	3431.430176	3697.653564	3243.665283	-0.014459	0.669075	-1.351062	453.609192	1.000000	7452.788086	0.705302	0.345873	1.798131	00:25
38	3417.913086	3756.547607	3317.378418	-0.013135	0.669685	-1.324888	438.777130	1.000000	7666.058594	0.692281	0.342241	1.745103	00:25
39	3409.144043	3647.353027	3204.307373	-0.020123	0.668135	-1.331397	442.705261	1.000000	7376.371582	0.697968	0.341519	1.750849	00:25
40	3407.758301	3649.263428	3219.372070	-0.006288	0.682669	-1.265153	429.512146	1.000000	7415.768066	0.700749	0.343829	1.706248	00:25
41	3382.909668	3686.329590	3223.107910	-0.011264	0.663852	-1.383487	462.748718	1.000000	7278.731445	0.707191	0.346626	1.832180	00:25
42	3376.441406	3638.436035	3196.129883	-0.002653	0.672465	-1.340442	441.959717	1.000000	7526.583008	0.681990	0.349581	1.793306	00:25
43	3373.004639	3626.475586	3170.622314	-0.014423	0.665697	-1.368459	455.530701	1.000000	7291.094727	0.698909	0.349142	1.801970	00:25
44	3366.769287	3631.925293	3173.363770	-0.018959	0.659315	-1.397636	458.319580	1.000000	7270.541016	0.691233	0.347236	1.822071	00:25
45	3347.360840	3574.262451	3142.636963	-0.013239	0.668440	-1.326434	431.249786	1.000000	7364.531738	0.674849	0.342823	1.737871	00:25
46	3317.150879	3598.666748	3156.432617	-0.001533	0.668835	-1.350455	441.818359	1.000000	7342.520020	0.678997	0.347351	1.794104	00:25
47	3323.658203	3565.412109	3115.931641	-0.008903	0.667702	-1.356261	449.135681	1.000000	7219.029785	0.692640	0.346469	1.800012	00:25
48	3311.372559	3567.005615	3113.006104	-0.010357	0.668052	-1.343055	453.667999	1.000000	7082.435059	0.711734	0.346697	1.768867	00:25
49	3321.950195	3582.604004	3113.802490	0.000333	0.656901	-1.400587	468.360321	1.000000	7102.209473	0.714338	0.346219	1.814144	00:26
50	3301.511719	3623.174561	3182.668213	0.008003	0.671113	-1.321199	440.130829	1.000000	7262.528320	0.694835	0.346230	1.740687	00:25
51	3285.044678	3552.255371	3106.276123	-0.006173	0.669649	-1.354133	445.727448	1.000000	7177.645996	0.682762	0.350433	1.800857	00:25
52	3286.866455	3675.446045	3236.316650	-0.013157	0.676845	-1.295927	438.723022	1.000000	7318.729980	0.703599	0.346884	1.728112	00:25
53	3291.747803	3667.880127	3209.201904	-0.017041	0.670350	-1.326245	458.201050	1.000000	7180.023926	0.729949	0.348613	1.740342	00:25
54	3284.672852	3593.201660	3139.171631	-0.014269	0.670780	-1.354988	453.624573	1.000000	7211.184570	0.697685	0.352816	1.806872	00:25
55	3267.846924	3536.163086	3085.779541	-0.002470	0.663568	-1.371893	450.007965	1.000000	7252.326660	0.686323	0.348915	1.796228	00:25
56	3250.682617	3564.268555	3101.536377	-0.012916	0.666759	-1.372767	462.398376	1.000000	7097.330566	0.708008	0.353114	1.815818	00:25
57	3254.443604	3547.814453	3098.741943	-0.006862	0.665436	-1.359174	448.660187	1.000000	7157.088379	0.691026	0.348435	1.784791	00:26
58	3260.725586	3530.732178	3079.662109	-0.003880	0.665441	-1.359928	450.669525	1.000000	7183.651855	0.694633	0.348688	1.782578	00:25
59	3236.969482	3484.071289	3034.793945	0.002332	0.670223	-1.346266	448.978607	1.000000	7092.309082	0.694828	0.351385	1.783404	00:25
60	3225.157715	3552.310791	3117.349854	-0.007228	0.672280	-1.309015	434.601807	1.000000	7282.853027	0.690033	0.345595	1.724956	00:25
61	3220.063232	3524.829834	3085.624756	-0.005145	0.671900	-1.325763	438.820679	1.000000	7248.909180	0.686005	0.349257	1.752840	00:26
62	3227.847168	3484.733643	3023.396729	-0.007903	0.657956	-1.388762	461.027283	1.000000	6973.873535	0.704548	0.348345	1.789521	00:26
63	3190.125000	3485.780029	3038.699463	-0.006577	0.668708	-1.349016	446.739380	1.000000	7149.849121	0.689440	0.351118	1.780027	00:25
64	3197.777832	3448.150146	2999.025879	-0.004879	0.659723	-1.374128	448.797913	1.000000	6996.356934	0.686545	0.346901	1.771503	00:25
65	3207.785889	3439.828369	2991.286133	0.004332	0.666682	-1.338493	448.244476	1.000000	6964.046387	0.704034	0.347361	1.744932	00:25
66	3216.496582	3504.475586	3048.498779	0.005608	0.656708	-1.394906	455.563385	1.000000	7071.123535	0.688872	0.348882	1.791947	00:25
67	3205.509766	3455.002930	3015.201904	-0.002899	0.671343	-1.323578	439.482635	1.000000	7103.756348	0.689148	0.350410	1.740655	00:25
68	3190.522461	3437.476318	2982.573242	-0.001924	0.664799	-1.361731	454.539734	1.000000	6959.311523	0.702032	0.350246	1.781555	00:25
69	3179.375000	3434.706055	2972.019043	-0.000864	0.661195	-1.389103	462.347748	1.000000	6931.499023	0.702403	0.351853	1.810729	00:25
70	3154.015137	3450.184814	3001.612061	-0.000603	0.654439	-1.383468	448.301758	1.000000	7242.174316	0.685155	0.344582	1.757106	00:26
71	3109.754395	3367.349121	2874.592285	0.014566	0.634897	-1.472259	492.536652	1.000000	6862.613281	0.737525	0.344089	1.801952	00:26
72	3043.930176	3312.779297	2833.916260	-0.007480	0.644100	-1.434898	478.643097	1.000000	6766.743652	0.723580	0.348020	1.781140	00:25
73	2990.204834	3322.297363	2830.795410	-0.008267	0.642356	-1.456952	491.218353	1.000000	6806.497559	0.736002	0.350011	1.814443	00:25
74	2982.461426	3225.157471	2743.953613	-0.002834	0.637180	-1.477048	481.043793	1.000000	6698.075684	0.704703	0.349975	1.816552	00:25
75	2936.034180	3200.522705	2732.647705	-0.000358	0.648334	-1.434253	467.748627	1.000000	6595.842285	0.693321	0.353818	1.802200	00:25
76	2917.427979	3188.411133	2711.284668	-0.010846	0.649576	-1.430211	476.969299	1.000000	6641.311035	0.715748	0.352910	1.806378	00:26
77	2900.190918	3208.352539	2738.323975	0.001015	0.643907	-1.449119	469.884857	1.000000	6799.942383	0.692973	0.351222	1.807197	00:25
78	2893.122559	3170.847412	2691.765869	-0.006294	0.643069	-1.466224	479.063080	1.000000	6557.207520	0.699490	0.355875	1.826500	00:25
79	2879.305420	3165.599121	2672.646484	-0.009947	0.638078	-1.504574	492.835205	1.000000	6531.899414	0.706808	0.357110	1.871682	00:26
80	2871.717529	3141.771240	2661.048340	0.008516	0.644238	-1.472341	480.623749	1.000000	6581.002441	0.697051	0.357159	1.850578	00:26
81	2841.466553	3114.589111	2620.065918	-0.013839	0.643850	-1.486288	494.445160	1.000000	6409.358398	0.716239	0.360313	1.870289	00:26
82	2827.166260	3109.616699	2619.346680	-0.004788	0.640471	-1.500353	490.193634	1.000000	6416.575684	0.700102	0.360305	1.870650	00:25
83	2806.600342	3085.924561	2605.619385	0.007370	0.646208	-1.465798	480.237457	1.000000	6450.090820	0.697967	0.358896	1.845136	00:26
84	2791.562256	3063.895020	2580.600586	-0.005374	0.645183	-1.463034	483.229340	1.000000	6309.436035	0.707763	0.358645	1.829669	00:25
85	2770.278076	3034.631836	2558.484863	-0.010086	0.647094	-1.460025	476.043915	1.000000	6346.354004	0.691674	0.359322	1.838968	00:25
86	2755.365479	3014.786133	2529.739014	-0.004383	0.644172	-1.489430	484.962555	1.000000	6343.630371	0.692092	0.361628	1.875817	00:26
87	2738.068115	3026.836182	2534.303467	-0.006969	0.645632	-1.497038	492.442505	1.000000	6328.986816	0.700723	0.364204	1.900079	00:25
88	2720.661865	2986.312256	2502.743164	-0.004747	0.646495	-1.490602	483.493469	1.000000	6292.840332	0.683937	0.365157	1.889861	00:25
89	2698.882568	2965.148682	2481.424072	-0.002050	0.646837	-1.491359	483.654999	1.000000	6172.981934	0.683138	0.365614	1.893745	00:25
90	2688.093262	2944.035889	2457.755615	-0.002745	0.643649	-1.513131	486.249237	1.000000	6184.668945	0.675225	0.367282	1.911071	00:25
91	2674.913086	2943.592041	2454.669922	-0.002578	0.644348	-1.508578	488.867065	1.000000	6166.262207	0.683654	0.367843	1.904162	00:25
92	2655.663086	2926.737305	2436.727295	-0.002487	0.646099	-1.506547	489.977570	1.000000	6111.576660	0.685432	0.368832	1.912587	00:26
93	2640.149414	2910.103271	2416.333984	-0.002656	0.642178	-1.537301	493.722076	1.000000	6126.336426	0.674277	0.370405	1.945017	00:25
94	2629.382812	2903.057617	2412.663330	0.002669	0.642730	-1.537003	490.318329	1.000000	6109.676270	0.665981	0.370631	1.949367	00:25
95	2615.815918	2896.590576	2403.968506	-0.000958	0.644573	-1.534148	492.563263	1.000000	6112.826172	0.670643	0.372265	1.953958	00:26
96	2607.981445	2885.615967	2391.821045	-0.003212	0.642997	-1.540947	493.727631	1.000000	6075.826660	0.669907	0.372110	1.956079	00:25
97	2609.968506	2881.287109	2386.099365	-0.002940	0.642660	-1.548294	495.115814	1.000000	6065.745605	0.667622	0.372756	1.968524	00:25
98	2603.308350	2880.835693	2387.826660	-0.004268	0.643977	-1.538081	492.949677	1.000000	6068.658203	0.669138	0.372360	1.957721	00:25
99	2598.858887	2880.998047	2387.606201	-0.004501	0.643797	-1.540458	493.339844	1.000000	6068.282227	0.668343	0.372554	1.961107	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	58740.539062	55176.195312	54889.167969	0.120437	0.523871	-1.432346	5821.192383	0.047932	60959.058594	2.827253	0.179028	0.830196	00:25
1	45920.312500	32378.673828	32121.812500	-0.037095	0.680219	-0.876800	1361.199707	0.185387	44998.371094	1.317362	0.198534	0.791697	00:25
2	28567.009766	18783.580078	18389.089844	-0.023480	0.572010	-1.268163	1013.169067	0.385156	32817.027344	0.967404	0.211317	0.809108	00:26
3	19321.156250	13977.159180	13471.999023	-0.019137	0.579537	-1.282815	827.127136	0.607671	27876.337891	0.787131	0.226196	0.940032	00:25
4	14469.906250	12062.826172	11523.424805	-0.031412	0.642168	-1.077186	664.454651	0.808860	25497.568359	0.700015	0.236848	0.963814	00:25
5	11711.579102	10398.442383	9849.642578	-0.026665	0.686315	-0.960859	576.726868	0.948876	23399.960938	0.636489	0.249260	1.035778	00:25
6	10033.545898	9363.882812	8784.699219	-0.025187	0.691321	-0.990074	577.364075	0.999986	21912.365234	0.602564	0.263317	1.168671	00:25
7	9010.625977	8700.817383	8128.997070	-0.021991	0.701658	-0.985699	570.353088	1.000000	21030.861328	0.579129	0.276201	1.233390	00:25
8	8411.709961	8434.400391	7861.457031	-0.025626	0.711409	-0.979375	571.588013	1.000000	20555.650391	0.569511	0.285582	1.291906	00:25
9	8147.944824	8266.574219	7697.082031	-0.025811	0.712803	-0.977301	568.101562	1.000000	20270.447266	0.564367	0.286426	1.297384	00:26

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	7651.367676	7365.542969	6727.640625	-0.001503	0.707371	-0.946704	636.378296	1.000000	18033.966797	0.688343	0.267682	1.189982	00:28
1	6716.003906	6474.808594	5877.452637	-0.036604	0.720139	-0.926342	595.863586	1.000000	16393.494141	0.638700	0.274323	1.235185	00:28
2	5997.029297	5860.405273	5274.698242	-0.018035	0.736100	-0.887804	584.582947	1.000000	15014.263672	0.634403	0.281981	1.247762	00:28
3	5485.274414	5364.577148	4783.366699	-0.004125	0.744810	-0.855897	580.197937	1.000000	13838.269531	0.644898	0.280916	1.228757	00:28
4	5094.323730	5222.379883	4581.991699	0.001645	0.730681	-0.953955	639.444214	1.000000	13211.501953	0.650297	0.294958	1.385436	00:28
5	4805.841309	4832.098145	4243.259277	-0.007232	0.739415	-0.903473	588.059143	1.000000	12172.434570	0.618609	0.291089	1.310845	00:28
6	4583.566406	4695.562012	4069.434326	0.008616	0.726104	-0.973470	625.385925	1.000000	11474.740234	0.621857	0.297524	1.394230	00:28
7	4416.959961	4745.934570	4140.187012	0.001199	0.740462	-0.918071	604.966614	1.000000	11435.058594	0.623564	0.298082	1.354706	00:28
8	4314.148438	4493.190918	3870.912598	-0.005136	0.733318	-0.939494	621.700745	1.000000	10547.795898	0.636232	0.296641	1.357139	00:28
9	4209.107910	4406.982422	3759.320068	-0.001849	0.727758	-1.000237	647.177917	1.000000	10461.577148	0.620679	0.307983	1.468478	00:28
10	4109.696777	4476.614258	3839.443359	-0.009990	0.732419	-0.969451	636.434265	1.000000	10169.422852	0.627681	0.304734	1.423959	00:28
11	4020.404785	4275.003906	3613.874756	-0.010094	0.728018	-1.003290	660.770203	1.000000	9553.526367	0.634482	0.308647	1.477842	00:28
12	3952.284912	4187.662598	3551.754150	0.005761	0.734981	-0.966541	635.446228	1.000000	9338.792969	0.623738	0.309741	1.428058	00:28
13	3900.007324	4116.317871	3462.226562	-0.006851	0.731415	-0.980627	653.731018	1.000000	8986.081055	0.639830	0.308371	1.442627	00:28
14	3851.263916	4204.222656	3555.328857	-0.012392	0.733164	-0.978597	648.536926	1.000000	9004.118164	0.632044	0.310562	1.446320	00:28
15	3785.325684	4007.054443	3348.741943	-0.000152	0.731587	-1.004563	658.063110	1.000000	8655.161133	0.622582	0.315733	1.497758	00:28
16	3767.278564	4183.640625	3517.904785	-0.008906	0.731445	-1.011055	665.384583	1.000000	8616.531250	0.626313	0.316394	1.513834	00:27
17	3758.536621	4103.028320	3443.043213	0.004418	0.725755	-1.007025	659.650574	1.000000	8621.183594	0.632305	0.310023	1.463498	00:28
18	3725.813477	3991.390381	3325.551514	-0.010483	0.731626	-1.013289	665.482483	1.000000	8288.931641	0.623968	0.317578	1.520346	00:28
19	3693.500732	3983.870850	3324.956543	0.013614	0.727927	-1.009194	658.678589	1.000000	8113.777832	0.625404	0.312643	1.483721	00:28
20	3654.010498	3962.879150	3288.293945	0.001136	0.722652	-1.044258	674.351501	1.000000	8213.806641	0.621986	0.315267	1.532625	00:28
21	3647.004395	3983.133789	3338.108398	0.004072	0.739812	-0.957298	644.710266	1.000000	8219.560547	0.635629	0.312068	1.439436	00:28
22	3617.129395	3887.343506	3229.427979	0.008313	0.730873	-1.007894	657.713196	1.000000	7895.292480	0.620299	0.315637	1.502362	00:28
23	3608.985107	3904.537842	3230.343018	0.000425	0.725290	-1.030399	674.001465	1.000000	7845.946289	0.629264	0.315511	1.516837	00:28
24	3591.437744	3859.909424	3193.618164	0.010155	0.734639	-1.009018	666.101379	1.000000	7719.414551	0.623905	0.318807	1.533824	00:28
25	3577.002197	3895.349121	3224.742920	-0.000528	0.729889	-1.021132	670.441528	1.000000	7743.109375	0.625357	0.318466	1.525949	00:28
26	3571.908447	3821.641602	3162.054688	0.000455	0.734451	-1.004335	659.434448	1.000000	7697.803711	0.619632	0.318668	1.517508	00:28
27	3548.191895	3849.494141	3180.026367	-0.004974	0.725175	-1.028136	669.249573	1.000000	7672.394043	0.626161	0.313801	1.516769	00:28
28	3533.383545	4059.682129	3408.733398	-0.000392	0.734781	-0.987317	650.586487	1.000000	8089.685059	0.623778	0.314767	1.479183	00:28
29	3516.743164	3760.632324	3109.681396	0.006230	0.734859	-0.989964	650.809082	1.000000	7584.377441	0.620539	0.317219	1.483018	00:28
30	3506.385498	3800.312500	3123.494629	-0.005099	0.726935	-1.031704	676.580688	1.000000	7519.890625	0.627771	0.318481	1.531833	00:28
31	3484.509277	3836.599365	3150.462402	-0.003075	0.719542	-1.075821	686.004456	1.000000	7661.994141	0.610878	0.322514	1.581861	00:28
32	3495.000732	3849.746582	3175.043701	0.010477	0.730324	-1.023581	674.504395	1.000000	7673.555176	0.626465	0.321434	1.533255	00:28
33	3487.385498	3826.780029	3129.293945	0.018080	0.733950	-1.026333	697.271606	1.000000	7434.991699	0.645661	0.325797	1.565082	00:28
34	3452.620361	3733.065918	3053.726318	0.004488	0.733624	-1.027590	679.199219	1.000000	7412.557617	0.622742	0.326465	1.560631	00:27
35	3455.627930	3729.029297	3067.841309	-0.000455	0.737738	-0.984265	660.988892	1.000000	7427.024414	0.632531	0.321030	1.482692	00:28
36	3448.309326	3716.426270	3055.599121	0.000706	0.732191	-1.013381	660.638794	1.000000	7478.499512	0.614729	0.322410	1.516803	00:28
37	3425.744873	3717.308594	3067.585449	0.009872	0.737455	-0.981283	649.612366	1.000000	7241.781738	0.621098	0.320518	1.472866	00:28
38	3406.716309	3655.440430	2981.873291	0.019556	0.730443	-1.028612	673.444092	1.000000	7264.535156	0.619335	0.324188	1.543146	00:28
39	3397.938721	3687.032959	2996.398438	0.017997	0.725269	-1.059344	690.533447	1.000000	7285.152832	0.619712	0.328644	1.572508	00:28
40	3396.500488	3704.924805	3025.917236	0.004099	0.733460	-1.027411	678.857178	1.000000	7252.134766	0.621408	0.328601	1.556395	00:28
41	3400.028320	3715.416748	3045.153564	0.004757	0.729243	-1.017588	670.118958	1.000000	7224.676270	0.626427	0.322481	1.500674	00:28
42	3389.938232	3709.358887	3035.591797	0.002872	0.730203	-1.026232	673.540894	1.000000	7237.694824	0.620630	0.326542	1.524723	00:28
43	3378.010010	3683.706787	2999.656250	0.010118	0.723173	-1.058739	683.975342	1.000000	7201.193359	0.614366	0.329194	1.549945	00:28
44	3378.157715	3711.697510	3017.652832	0.010882	0.726414	-1.050995	693.858459	1.000000	7120.746094	0.630194	0.327425	1.563918	00:27
45	3371.152588	3661.925781	2982.276611	0.002141	0.728925	-1.028453	679.527405	1.000000	7222.122070	0.628844	0.324787	1.527632	00:28
46	3362.568604	3678.501709	3016.866455	0.002026	0.734364	-1.003709	661.490845	1.000000	7162.108887	0.619610	0.324967	1.503220	00:28
47	3345.114502	3662.004883	3013.256836	0.007719	0.736944	-0.981593	648.579651	1.000000	7179.213867	0.619943	0.321520	1.468639	00:28
48	3339.086670	3722.939941	3025.527588	0.014281	0.727889	-1.059957	697.148499	1.000000	7148.197266	0.623049	0.331727	1.594466	00:28
49	3337.108643	3629.774658	2950.814941	0.008032	0.728444	-1.030823	678.777283	1.000000	7144.182617	0.625660	0.326432	1.524955	00:28
50	3311.899414	3642.291748	2961.773438	0.001066	0.730269	-1.038538	680.423950	1.000000	7062.062500	0.617276	0.330441	1.555513	00:28
51	3320.456543	3581.983154	2891.198242	0.006517	0.728271	-1.043807	690.645813	1.000000	6893.710449	0.627845	0.331142	1.549931	00:28
52	3307.604736	3677.112793	3016.974121	0.007375	0.736529	-0.997537	659.946533	1.000000	7134.888672	0.619345	0.326338	1.500275	00:28
53	3312.864990	3641.685059	2937.004639	-0.004953	0.720087	-1.083988	704.637512	1.000000	6979.736328	0.622097	0.331982	1.590897	00:28
54	3296.139648	3577.402588	2884.481689	0.004521	0.728107	-1.051127	692.779541	1.000000	7009.988770	0.624056	0.332423	1.568000	00:28
55	3282.488037	3654.056396	2947.325928	0.014008	0.723545	-1.081236	706.626526	1.000000	7055.623535	0.621703	0.333729	1.613418	00:28
56	3289.469238	3602.896240	2920.512939	0.006522	0.728315	-1.034476	682.367432	1.000000	6979.327637	0.626334	0.328607	1.531155	00:28
57	3292.510498	3604.759033	2907.411865	0.001527	0.724153	-1.068953	697.232239	1.000000	7029.240234	0.619716	0.333099	1.584040	00:28
58	3279.932373	3631.697266	2934.322021	0.013448	0.722786	-1.074533	697.274597	1.000000	6931.425781	0.616444	0.333294	1.585446	00:28
59	3263.800537	3766.281738	3084.604980	0.010203	0.727060	-1.042177	681.429016	1.000000	7250.323730	0.619173	0.330562	1.536164	00:28
60	3247.138428	3614.538818	2906.910400	0.012615	0.722017	-1.082184	707.588440	1.000000	6928.161621	0.622702	0.336169	1.592687	00:28
61	3261.955566	3540.338135	2835.608398	0.003212	0.724194	-1.073995	704.604065	1.000000	6790.403320	0.623425	0.335638	1.589854	00:27
62	3256.552002	3535.280029	2826.410645	0.001747	0.721265	-1.087617	708.741272	1.000000	6755.615723	0.621381	0.335210	1.605481	00:28
63	3250.792480	3548.330078	2842.145508	0.018147	0.726087	-1.072173	706.068848	1.000000	6790.458984	0.624347	0.335631	1.605922	00:28
64	3261.629150	3514.951660	2814.265869	0.005941	0.722353	-1.069727	700.618652	1.000000	6820.963379	0.625080	0.333472	1.567700	00:28
65	3241.566650	3608.032715	2904.027832	0.006998	0.723039	-1.076644	703.917480	1.000000	6986.800781	0.622058	0.335172	1.589766	00:28
66	3224.029053	3570.575195	2885.435303	0.016744	0.725186	-1.057620	685.048767	1.000000	6912.272949	0.612871	0.331627	1.565044	00:28
67	3251.290039	3583.719482	2884.775635	0.011005	0.723794	-1.060478	698.897400	1.000000	6890.466797	0.629185	0.332542	1.557823	00:28
68	3238.050049	3528.008789	2828.175049	0.019224	0.726878	-1.064695	699.730469	1.000000	6830.160645	0.621388	0.335719	1.589031	00:28
69	3223.922852	3530.111084	2814.860352	0.006388	0.723904	-1.079526	715.156250	1.000000	6741.392090	0.631759	0.336924	1.602475	00:28
70	3216.445068	3500.061523	2782.776855	0.008921	0.720420	-1.104010	717.167419	1.000000	6705.318359	0.617773	0.338750	1.635759	00:29
71	3207.145996	3521.456787	2817.231201	0.005418	0.720238	-1.086549	704.176575	1.000000	6791.708496	0.616982	0.336408	1.589192	00:28
72	3204.608154	3531.666260	2808.478760	0.009166	0.717051	-1.115761	723.127869	1.000000	6687.869629	0.620199	0.337613	1.643584	00:28
73	3213.811523	3598.167236	2889.819580	0.010586	0.726090	-1.083778	708.206543	1.000000	6976.821777	0.615494	0.339968	1.628908	00:28
74	3203.632324	3670.336182	2951.901855	0.005238	0.719610	-1.084969	718.238342	1.000000	6905.762695	0.637379	0.334413	1.585946	00:28
75	3207.873535	3579.372803	2876.421875	0.006835	0.723253	-1.067832	702.820068	1.000000	6790.896484	0.627843	0.334451	1.570162	00:28
76	3187.589355	3531.233643	2829.830811	0.013975	0.722816	-1.077619	701.346497	1.000000	6883.243164	0.618706	0.334157	1.596420	00:28
77	3181.636719	3545.495117	2852.156006	0.002698	0.731067	-1.049801	693.244995	1.000000	6794.460449	0.620472	0.335831	1.587278	00:28
78	3177.850098	3536.807861	2822.090088	0.005285	0.719301	-1.111130	714.644043	1.000000	6781.951172	0.608447	0.341650	1.637716	00:28
79	3180.280518	3572.795410	2873.986572	0.006143	0.722267	-1.085594	698.616516	1.000000	6843.494141	0.607717	0.337304	1.603625	00:28
80	3168.005371	3535.424561	2833.231201	0.019834	0.726783	-1.064308	702.152222	1.000000	6706.426758	0.623918	0.337304	1.585141	00:28
81	3152.529297	3446.023438	2734.945557	0.008887	0.723976	-1.088619	710.974243	1.000000	6700.422852	0.617348	0.340108	1.623608	00:28
82	3135.922119	3434.467773	2714.553223	0.012614	0.720932	-1.108265	719.850525	1.000000	6591.262695	0.614366	0.343730	1.639695	00:27
83	3113.646729	3466.451172	2765.960205	0.003383	0.726390	-1.076803	700.406921	1.000000	6657.192871	0.610935	0.339992	1.610990	00:28
84	3098.293701	3459.226318	2755.147705	0.016013	0.724201	-1.087780	703.987000	1.000000	6683.789551	0.607959	0.341545	1.616467	00:28
85	3087.032959	3389.620605	2680.396240	0.007750	0.721710	-1.108647	709.155762	1.000000	6485.849121	0.599022	0.344326	1.646617	00:28
86	3070.500977	3382.860840	2672.243164	0.014384	0.723655	-1.103893	710.518250	1.000000	6548.608398	0.602184	0.344518	1.650620	00:28
87	3055.568359	3381.640869	2666.888916	0.006845	0.725388	-1.102032	714.671509	1.000000	6474.822754	0.606631	0.345363	1.659149	00:28
88	3045.261719	3402.307373	2680.911377	0.006631	0.723334	-1.119401	721.316833	1.000000	6557.485352	0.602261	0.347152	1.684888	00:28
89	3029.100830	3327.909180	2613.521240	0.009250	0.723729	-1.116796	714.319580	1.000000	6478.024902	0.594214	0.348252	1.677276	00:28
90	3009.680908	3317.802979	2615.427002	0.011575	0.728394	-1.091922	702.291748	1.000000	6491.918945	0.594770	0.345983	1.656690	00:28
91	2996.286865	3309.975342	2594.326660	0.006130	0.726961	-1.113291	715.553467	1.000000	6424.102539	0.593813	0.349996	1.692982	00:28
92	2977.406250	3292.775391	2571.700195	0.013427	0.725092	-1.122516	721.000305	1.000000	6330.194824	0.594845	0.350828	1.698746	00:27
93	2972.783447	3268.155273	2544.673096	0.008637	0.725328	-1.129471	723.458557	1.000000	6335.770508	0.591184	0.352287	1.717149	00:28
94	2970.029785	3270.115967	2551.228760	0.009532	0.724969	-1.130088	718.871765	1.000000	6373.247559	0.585079	0.352135	1.716223	00:28
95	2954.962891	3261.302979	2542.979004	0.008221	0.725357	-1.124736	718.260071	1.000000	6304.031250	0.588493	0.351823	1.705143	00:28
96	2950.425049	3252.043213	2532.653076	0.010782	0.725520	-1.130057	719.378235	1.000000	6317.508301	0.584415	0.353175	1.717738	00:28
97	2943.955322	3251.385254	2532.198486	0.008705	0.726147	-1.125400	719.156494	1.000000	6292.142090	0.587820	0.352392	1.712947	00:28
98	2944.449463	3245.879883	2527.619141	0.008804	0.725314	-1.130287	718.228455	1.000000	6309.519043	0.583112	0.352947	1.717500	00:28
99	2937.538330	3244.074463	2529.292725	0.008952	0.725937	-1.125587	714.744995	1.000000	6312.990234	0.581985	0.352475	1.711296	00:28

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	40959.515625	37905.636719	37891.761719	8.208526	-0.024008	0.321359	-2.330088	51717.753906	0.548548	277.020477	558.442444	0.462210	00:23
1	28461.808594	31310.380859	31296.017578	11.071305	-0.001257	0.356824	-2.144540	45347.582031	0.634957	318.794250	540.526062	0.561912	00:24
2	20569.023438	24270.050781	24254.761719	12.098309	-0.021030	0.391993	-1.954344	39474.792969	0.675133	337.234344	506.135071	0.548326	00:23
3	14445.319336	15024.644531	15008.929688	13.079365	-0.020070	0.419365	-1.815197	30104.261719	0.710429	353.417725	484.618500	0.530492	00:24
4	10298.541016	9405.922852	9392.183594	12.310252	-0.012689	0.445155	-1.688505	22990.738281	0.714922	355.908539	453.056366	0.504430	00:24
5	7728.354980	6665.394531	6650.482422	13.403973	-0.013079	0.452304	-1.656670	18389.359375	0.737011	364.198914	455.659149	0.502500	00:23
6	6190.182129	5603.939941	5589.669434	12.933229	-0.013793	0.479955	-1.534696	15767.240234	0.748866	369.001404	430.846649	0.491857	00:23
7	5236.472656	5212.584961	5196.565430	14.607766	-0.016758	0.491669	-1.485687	14362.958984	0.785005	385.483612	437.185181	0.486615	00:23
8	4617.360840	4473.277344	4458.379395	14.115689	-0.030596	0.498461	-1.451891	12745.385742	0.778319	385.171143	424.746368	0.459398	00:23
9	4208.974609	4216.414551	4200.229004	15.131713	-0.027884	0.513088	-1.392592	11635.668945	0.803424	394.603180	423.435181	0.455078	00:23
10	3901.279785	4004.926025	3988.525146	15.441641	-0.022382	0.523603	-1.348659	10846.424805	0.816376	402.754547	419.191315	0.437780	00:24
11	3670.582275	3740.616455	3724.684570	15.171966	-0.032123	0.524357	-1.344947	10166.298828	0.812432	399.097137	416.266876	0.438142	00:24
12	3491.620117	3667.437744	3651.875488	14.801250	-0.026936	0.534644	-1.305375	9781.682617	0.815828	401.885956	408.844971	0.435637	00:24
13	3373.058594	3597.774658	3581.469238	15.714341	-0.029027	0.541989	-1.278454	9533.083984	0.833115	409.734467	412.325806	0.437646	00:23
14	3268.388672	3494.202637	3478.181396	15.434155	-0.029170	0.551247	-1.245216	9079.128906	0.836138	408.363190	406.691254	0.440574	00:24
15	3192.942139	3778.173584	3761.590820	15.698897	-0.017824	0.561030	-1.210492	9206.311523	0.848363	414.858185	405.788605	0.443565	00:24
16	3132.410400	3364.331055	3348.152100	15.786729	-0.019340	0.569641	-1.177485	8492.832031	0.853677	417.495789	401.234131	0.430003	00:23
17	3073.229492	3397.922852	3381.511230	15.826709	-0.029253	0.571797	-1.171749	8525.550781	0.855620	416.931549	401.589661	0.439019	00:24
18	3024.521240	3429.331299	3412.252441	16.369137	-0.021574	0.587442	-1.115198	8408.209961	0.877177	429.384705	401.066406	0.427119	00:23
19	2968.729004	3244.765137	3229.118896	15.019815	-0.023893	0.582480	-1.132304	8225.012695	0.854739	416.707214	391.953735	0.428889	00:23
20	2925.205078	3132.208984	3115.920166	16.067533	-0.026519	0.596821	-1.084884	7874.996582	0.875232	426.971710	394.178131	0.432377	00:24
21	2889.247314	3118.166748	3102.901123	15.002897	-0.029995	0.585165	-1.121761	7893.246582	0.851289	415.738831	387.821686	0.424242	00:24
22	2845.302734	3130.704346	3114.590576	15.593694	-0.020829	0.592054	-1.099054	7766.731934	0.867235	421.982574	392.017334	0.426566	00:24
23	2807.441162	3189.631836	3173.846924	14.906292	-0.021064	0.596690	-1.083462	7741.119629	0.862720	421.786804	386.648254	0.426360	00:23
24	2784.897949	3024.525879	3008.107178	15.926530	-0.020563	0.606408	-1.051424	7504.205566	0.881879	429.748260	391.035858	0.426201	00:23
25	2770.114746	3029.304199	3013.619873	15.341253	-0.023551	0.608746	-1.042673	7513.146973	0.875445	425.717957	385.517975	0.421918	00:24
26	2761.279541	3091.217041	3074.397949	16.858736	-0.025576	0.628417	-0.979442	7453.478027	0.908650	441.615448	392.550476	0.421370	00:24
27	2730.678223	2962.628418	2947.173584	15.276114	-0.026963	0.620887	-1.002515	7304.295410	0.881647	428.941559	381.008789	0.418781	00:24
28	2711.719971	2964.291016	2948.281006	15.699995	-0.030957	0.629126	-0.976302	7354.960449	0.894260	436.180389	383.461853	0.418827	00:24
29	2698.613770	3050.802002	3035.411377	14.809456	-0.014705	0.621483	-1.001865	7517.462891	0.878165	427.194489	378.995758	0.423432	00:23
30	2680.392822	2954.147461	2937.375244	16.296328	-0.016200	0.636249	-0.954343	7354.759277	0.910993	443.337250	388.900665	0.419872	00:23
31	2648.716309	3001.396973	2986.461182	14.623203	-0.042457	0.618944	-1.007994	7513.009766	0.869564	422.928467	375.671539	0.415266	00:23
32	2552.094238	2815.679688	2799.725830	15.741020	-0.021439	0.625477	-0.987615	7107.967773	0.892987	435.489410	384.390106	0.415763	00:23
33	2500.885742	2701.386475	2685.672852	15.191346	-0.037839	0.645597	-0.924251	6860.041016	0.902515	438.233551	378.975342	0.412627	00:24
34	2422.822266	2719.418457	2703.131348	16.222771	-0.024178	0.639103	-0.945698	6888.277344	0.908967	442.192963	386.675140	0.419654	00:23
35	2360.017090	2604.458252	2588.693359	15.577497	-0.019458	0.639788	-0.944168	6656.376465	0.902958	438.855408	382.952759	0.422265	00:23
36	2323.858398	2629.382080	2613.721191	15.710530	-0.040843	0.643459	-0.932621	6623.768066	0.904269	438.465942	382.072937	0.420569	00:23
37	2305.494873	2616.138916	2600.468262	15.508165	-0.032108	0.642783	-0.933116	6720.474121	0.901037	437.298370	379.680634	0.414019	00:23
38	2282.103271	2587.460205	2572.022217	15.235188	-0.030462	0.647844	-0.918168	6601.239258	0.902206	437.860565	377.771149	0.416694	00:24
39	2269.184570	2588.785400	2572.932373	15.794830	-0.031420	0.647192	-0.920478	6507.833008	0.909731	442.258911	382.598511	0.418556	00:23

OVERVIEW: cvae module - convolutional variational auto encoder

sneaker image encoding via various flavors of auto-encoder.

FUTURE EXTENSIONS:

Using fastai V2: data pipelining

prep_df_for_datablocks[source]

get_ae_btfms[source]

get_ae_no_aug[source]

FastAI "block" for autoencoder

class TensorPoint[source]

class Tensor2Vect[source]

class LatentsTensor[source]

df_get_x[source]

df_get_y[source]

LatentsTensorBlock[source]

df_ae_x[source]

df_ae_y[source]

LatentTupleBlock[source]

get_ae_DataBlock[source]

creating the VAE

AE: deterministic AutoEncoder ( non- variational)

class UpsampleBlock[source]

class LatentLayer[source]

class AEEncoder[source]

class AEDecoder[source]

build_AE_encoder[source]

build_AE_decoder[source]

class AE[source]

Loss

class AELoss[source]

Metrics

class MyMetric[source]

class L1LatentReg[source]

class KLDiv[source]

class L2MeanMetric[source]

class L1MeanMetric[source]

class L2Metric[source]

class L1Metric[source]

class L2BMeanMetric[source]

class L1BMeanMetric[source]

class KLWeightMetric[source]

class RawKLDMetric[source]

class WeightedKLDMetric[source]

class MuMetric[source]

class MuSDMetric[source]

class StdMetric[source]

class StdSDMetric[source]

class LogvarMetric[source]

class LogvarSDMetric[source]

metrics helpers

default_AE_metrics[source]

short_AE_metrics[source]

Callback - AnnealedLoss

class AnnealedLossCallback[source]

default_KL_anneal_in[source]

parameter Splitters

bn_splitter[source]

resnetVAE_split[source]

AE_split[source]

other encoder and decoder types

get_conv_parts[source]

get_pretrained_parts[source]

get_encoder_parts[source]

constructing VAE with Module Class Layers

class VAELinear[source]

class VAELayer[source]

simple BVAE class from VAE layer class components

class BVAE[source]

get_pretrained_parts[source]

VAE Loss functions classes

class BVAELoss[source]

default_VAE_metrics[source]

short_VAE_metrics[source]

"Vanilla" convolutional beta-VAE

MMD-VAE

class MMDVAE[source]

gaussian_kernel[source]

class MaxMeanDiscrepancy[source]

class MMDLoss[source]

class MMDMetric[source]

short_MMEVAE_metrics[source]

`prep_df_for_datablocks`[source]

`get_ae_btfms`[source]

`get_ae_no_aug`[source]

`class` `TensorPoint`[source]

`class` `Tensor2Vect`[source]

`class` `LatentsTensor`[source]

`df_get_x`[source]

`df_get_y`[source]

`LatentsTensorBlock`[source]

`df_ae_x`[source]

`df_ae_y`[source]

`LatentTupleBlock`[source]

`get_ae_DataBlock`[source]

`class` `UpsampleBlock`[source]

`class` `LatentLayer`[source]

`class` `AEEncoder`[source]

`class` `AEDecoder`[source]

`build_AE_encoder`[source]

`build_AE_decoder`[source]

`class` `AE`[source]

`class` `AELoss`[source]

`class` `MyMetric`[source]

`class` `L1LatentReg`[source]

`class` `KLDiv`[source]

`class` `L2MeanMetric`[source]

`class` `L1MeanMetric`[source]

`class` `L2Metric`[source]

`class` `L1Metric`[source]

`class` `L2BMeanMetric`[source]

`class` `L1BMeanMetric`[source]

`class` `KLWeightMetric`[source]

`class` `RawKLDMetric`[source]

`class` `WeightedKLDMetric`[source]

`class` `MuMetric`[source]

`class` `MuSDMetric`[source]

`class` `StdMetric`[source]

`class` `StdSDMetric`[source]

`class` `LogvarMetric`[source]

`class` `LogvarSDMetric`[source]

`default_AE_metrics`[source]

`short_AE_metrics`[source]

`class` `AnnealedLossCallback`[source]

`default_KL_anneal_in`[source]

parameter `Splitters`

`bn_splitter`[source]

`resnetVAE_split`[source]

`AE_split`[source]

`get_conv_parts`[source]

`get_pretrained_parts`[source]

`get_encoder_parts`[source]

`class` `VAELinear`[source]

`class` `VAELayer`[source]

`class` `BVAE`[source]

`get_pretrained_parts`[source]

`class` `BVAELoss`[source]

`default_VAE_metrics`[source]

`short_VAE_metrics`[source]

`class` `MMDVAE`[source]

`gaussian_kernel`[source]

`class` `MaxMeanDiscrepancy`[source]

`class` `MMDLoss`[source]

`class` `MMDMetric`[source]

`short_MMEVAE_metrics`[source]

`default_MMEVAE_metrics`[source]

`class` `UpsampleResBlock`[source]

`get_resblockencoder_parts`[source]

`class` `ResBlockAEDecoder`[source]

`build_ResBlockAE_decoder`[source]

`class` `ResBlockAE`[source]